How We Generate SSL Certificates for Custom Domains using Let’s Encrypt

Dom Harrington -

The Problem

Here at ReadMe, when you create a docs site it is automatically given a URL of project-name.readme.io. While this is great for smaller company APIs and open source projects, it can make your developer hub seem more professional and trustworthy if it has a consistent URL with your product: docs.example.com. Currently, enabling SSL involves the following steps:

  • the user purchases an SSL certificate ($$$);
  • that certificate needs to be uploaded to our site;
  • we create a new Heroku app to upload your certificate to ($20 per certificate, per month);
  • Heroku gives us a random URL for this app (something like tochigi-1877.herokussl.com ;
  • you have to CNAME to this endpoint;
  • we have to add your custom domain to our main Heroku app so that your certificate can be served.

This process has many moving parts, caused a lot of support and is expensive for us (we charge $59/mo, and $20/mo of that goes to SSL). Our Heroku bill is in the double-digit thousands. Heroku has recently launched their free SSL certificate beta which utilizes SNI to be able to serve multiple SSL certificates per IP, however it was still a bunch of extra steps for the end user.

Our requirements for a replacement:

  • Simpler setup for our users—we wanted it to just work.
  • Low cost.
  • Supports SNI (so that we can serve many SSL certificates from one IP).

Let's Encrypt

I’ve heard a lot about Let’s Encrypt since it came out back in April 2016. Let’s Encrypt is a certificate authority with a few differences: It only issues comparatively short-lived certificates (90 days); it has an API to allow programmatic renewals and most importantly—it’s free! The short lived certificates sound like a disadvantage on the surface but this is done to encourage developers to completely automate the certificate generation process and prevent it from being a step which happens once every 3 years by a developer who has probably forgotten the process.

Let’s Encrypt does have a few rate limits:

  1. 20 certificates per registered domain, e.g for the domain example.com you could register the certificates [1 - 20].example.com in one week. As we’re planning on generating certificates for domains other than our own, this should not be a problem.
  2. Duplicate certificate limit of 5 per week—this should not be a problem if we’re only requesting certificates for previously unseen domains.
  3. 500 registrations per IP address per 3 hours—Let’s Encrypt themselves say this one is pretty rare to hit so I’m not too concerned.

With the theoretical viability of LE confirmed, I decided to do some testing on a fresh EC2 instance using the official command line program certbot and this excellent guide from Digital Ocean. Within a couple of hours I managed to get it setup with a valid SSL certificate and automatic renewals via a daily cron job—sweet!

The one problem with using NGINX is that there does not appear to be a way to dynamically serve and generate SSL certificates natively for arbitrary domains. The directives you need to use ssl_certificate and ssl_certificate_key require paths to files that exist and NGINX will error on startup if those files do not exist. After some frantic googling I found a potential solution to this problem here.

This module promises to “provide on-the-fly SSL registration and renewal with OpenResty/nginx and Let’s Encrypt”. Wait … what’s OpenResty? OpenResty is a “dynamic web platform based on NGINX and LuaJIT”. This allows us to build logic into our nginx configuration files using Lua.

The next step was building out a proof of concept to see if this could work. OpenResty’s installation instructions leave a lot to be desired but fortunately it publishes an official docker image which should allow me to get setup in no time.

The lua-resty-auto-ssl module is an OpenResty plugin which uses the turbo-charged ssl_certificate_by_lua_block to provide the dynamic functionality. Here’s a reduced NGINX configuration example demonstrating this:

http {
  init_by_lua_block {
    auto_ssl = require("resty.auto-ssl").new()

    auto_ssl:set("allow_domain", function(domain)
    -- Insert your custom domain logic here
    end)

    auto_ssl:init()
  }

  init_worker_by_lua_block {
    auto_ssl:init_worker()
  }

  # HTTPS server
  server {
    listen 443 ssl;

    ssl_certificate_by_lua_block {
      auto_ssl:ssl_certificate()
    }
  }
}

You have to provide your own logic to determine whether to allow a domain through or not. This can be a static example such as a regular expression:

auto_ssl:set("allow_domain", function(domain)
  return ngx.re.match(domain, "^(example.com|example.net)$", "ijo")
end)

There is no way for us to statically determine our domains in this manner as the possibilities of customer’s domains are endless. Thankfully there is a module for OpenResty which allows us to make HTTP requests: https://github.com/pintsized/lua-resty-http. So I created an API endpoint in our node app which either returns a 200 or a 404 for a given domain to determine whether it is valid or not.

auto_ssl:set("allow_domain", function (domain)
  local http = require("resty.http")
  local httpc = http.new()

  httpc:set_timeout(5000)

  local uri = os.getenv("ALLOW_DOMAIN_API")..domain
  print("Querying readme api for custom domain: ", uri)

  local res, err = httpc:request_uri(uri, {
    method = "GET"
  })

  if not res then
    print("failed to request: ", err)
    return false
  end

  if res.status == 200 then
    print("Domain is allowed! Status code: ", res.status, " _id: ", res.body)
    return true
  end

  if res.status == 404 then
    print("Domain not found. Status code: ", res.status)
    return false
  end

  print("Unexpected response from API. Status code: ", res.status)
  return false
end)

The first time a user accesses with a new domain, we check to see if that domain is in our allowed list by querying our API. If it is, whilst the initial HTTP request is still open, a certificate is requested from Let’s Encrypt then transparently returned to the user. All subsequent requests are returned with this SSL certificate until it is ready to be renewed.

Results

One of our goals for a new system was to simplify the CNAME DNS setup for our customers. Now that we have a central server which supports SNI this is as easy as CNAMEing to ssl.readmessl.com and entering your custom domain into our dashboard.
This functionality is working well in testing and will be rolled out to production behind a flag in the coming weeks with a more general roll out once we’re confident it is working effectively. If you’d like to be one of the first users to test it then email support@readme.io and ask to have the flag turned on.

Donating

Switching to Let's Encrypt is going to save us (and our customers!) thousands of dollars every month. Let's Encrypt is a free alternative, however they run on donations. So, we're committed to donating $1/month for every cert they generate for us. We just made our first donation, and will be doing it monthly. If your company benefits from Let's Encrypt, donate!