I originally planned on writing a sales pitch for Let’s Encrypt and Certbot here, to serve as introduction. I’ll cut straight to the facts, instead:

• TLS certificates are GOOD. They prevent MITM attacks through unsecure WiFi and compromised networking equipment¹.
• Let’s Encrypt provides a production-ready tool, Certbot, to obtain TLS certificates. It’s free, reliable, and made with automation in mind.
• Certbot integrates easily with existing DevOps tooling. You don’t even have to run certbot, it runs itself; it even automatically renews the certificates periodically, so you no longer have to wake up to TLS errors on your website.

If the stakeholders require using Extended Validation (EV) certificates then you can’t use Certbot, but for any other case it’s my first option nowadays.

The default use case

In order to use Certbot:

1. Visit https://certbot.eff.org/
2. Select your web server or reverse proxy software you’re using, and the operating system or hosting provider that software is running on.
3. Follow the instructions

The instructions vary, obvously, but for the common use case of having a Nginx/Apache/HAProxy instance running on a server it involves either:

• a) using the appropriate plugin to serve a randomly generated file using the running web server.
• b) using the standalone install method, which binds itself to port 80 to serve the file.

This workflow is simple and performant for Internet exposed servers that aren’t behind a load-balancer, which is good enough for most applications out there.

Our use case

Time ago we wrote a blog post about HTTP reverse proxies over the Internet.

As shown in the diagram, there shouldn’t be any issues issuing certificates: example.com points to the proxy, and asdf.noip.com points to our homeserver.

But let’s look at the following real-world use case of mine:

• When I’m outside home I to connect to https://git.sinenie.cl, this will hit the reverse proxy due to our setup. Everything works OK.
• When I’m at home, however, I’ll change things so git.sinenie.cl resolves to the private IP address of my homeserver (e.g. 192.168.0.100), so I can get sub 1ms latencies, and speeds as fast as my cables/NIC/switches allow.

How do I get a certificate for git.sinenie.cl in my homeserver now?

The problem

If we attempt to use the default workflow in this case, Let’s Encrypt will hit the remote reverse proxy, instead of our homeserver.

And while this may seem a bit far-fetched, this isn’t the only scenario where you may hit this issue. Besides, if you want wildcard certificates you’ll have to use an alternative authenticator.

Candidate solutions

After thinking a bit, I converged into these 3 ideas:

1. Copy the certificate from the reverse proxy server to our homeserver.
2. Centralize certificate creation and renewal and push the certificates to each server.
3. Continue using Certbot on all our servers, but use the DNS authenticator plugins for the dns-01 challenge, instead of the default plugins for the http-01 challenge.

We’ll analyze each of these in more detail now.

1. Copy the certificate from the proxy server

Of course. The first thing to come to mind is to copy the files into our local server.

You just need to write a script that does the following:

1. Uses scp or rsync to download the Certbot directory and all of its contents into your filesystem.
2. Reloads services that cache certificates, like Nginx. e.g. systemctl reload nginx.

Yes, you have to create a new user on the proxy server and configure permissions so the homeserver can connect and sudo passwordlessly, but with Ansible it’s easy enough.

Besides that, there are other considerations regarding how Certbot uses symbolic links inside /etc/letsencrypt/live/, but if you replicate the directory structure you shouldn’t have issues with those links.

After you get the script working, it’s just a matter of adding a new cronjob, maybe daily, to ensure the homeserver has the latest certificates.

ADVANTAGES:

• Easy to understand.
• Relatively easy to implement.

DISADVANTAGES:

• You’ll always need another Internet-exposed server to get certificates of any kind. Even if that server doesn’t actually use the certificates.
• “Simple” bash scripts tend to become “less simple” as time goes on.
• Certbot already provides a more robust alternative that takes less work to use (DNS authenticator plugins).

2. Centralize certificate creation and renewal

Instead of pulling certificates from remote servers, we have a single server that pushes the certificates into them. The opposite of the previous option.

Actually, the comparison goes beyond that: if previously we wanted to solve our specific problem and forget about it, this is mostly about solving a problem we’ve yet to hit: rate limits.

Implementing this is a bit more involved, so we’ll leave it for a future blog post.

ADVANTAGES:

• Credentials and other sensitive information required for certificate renewal exist in one server, instead of many.
• It’s possible to create and dispose VPS instances freely, without worrying about false positive expiry alerts due to forgetting to non-revoked certs.
• Less likely to hit Let’s Encrypt rate limits, especially when compared to having a Certbot installation in every single server that you manage.

DISADVANTAGES:

• More complex than the alternatives.
• Potentially a single point of failure.

A detour into rate limits

Despite it being a free service, Let’s Encrypt has very generous rate limits.

We care mostly about two in particular:

• 50 Certificates per Registered Domain per week means that if you own example.com you cannot get create than 50 certificates for example.com or subdomains of it in a single week.
• 5 Duplicate Certificates per week means that you cannot create more than 5 certificates with the exact set of FQDN in a single week.

A renewal counts as a duplicate certificate. i.e. you cannot create more than 6 identical certificates in a given week, and you cannot renew more than 5 identical certificates in a given week.

Ref.: https://letsencrypt.org/docs/rate-limits/.

3. Use the DNS authenticator plugins

The DNS authenticator plugins use the dns-01 challenge, which requires a TXT record to be added to the domain’s DNS server.

If you’d like to obtain a wildcard certificate from Let’s Encrypt or run certbot on a machine other than your target webserver, you can use one of Certbot’s DNS plugins.
Ref.: https://certbot.eff.org/docs/using.html#dns-plugins

While this can be done manually, there are optional DNS plugins that can be used to automate the process.

Also, as mentioned earlier, it’s only possible to acquire wildcard certificates with the dns-01 challenge, probably because it suggests ownership over the entire domain’s DNS records, instead of a single host like it’s proven for the http-01 challenge.

ADVANTAGES:

• You don’t need to stop the webserver as it happens with the standalone authenticator.
• The server doesn’t need to be Internet-exposed.
• You can get wildcard certificates (e.g. *.example.com).
• Install a single package with the corresponding DNS plugin and you’re ready to go.

DISADVANTAGES:

• You have to install an additional package for the DNS plugin.
• You’re still bound to hit rate limits if you a have a lot of small servers hosting services with the same FQDN.
• You’re often leaving an API key or some other sort of credentials in the server for cert renew purposes.
  • Honestly though, if an attacker already got remote root access to your server you’re already fucked anyways lmao.

Solution implementation

I already use Ansible extensively for my own server provisioning needs, but I didn’t find an already existing role that fulfilled the need I described.

Therefore, I wrote my own with the following scope:

1. Support for RHEL/CentOS 8+.
2. Installs Certbot using Snap, which is the current recommended method.
3. Uses the DigitalOcean DNS plugin.
4. Create new certificates.
5. Setup auto-renewal using SystemD Timers or cron.

The source code is available at: https://git.sinenie.cl/max/ansible-role-certbot-dns

It’s not packaged yet, that’s why it’s missing an “Install” section in the README. For now cloning the repo to and symlinking it to ~/.ansible/roles works.

Summary

I wanted to share the role I wrote in the off-chance someone finds it useful, while also describing the problem and potential solutions comparison that led to me writing it.

Finally, there was something said about the [alleged] advantages of centralized certificate management, but it was out of scope for this post.

We’ll put that to the test on Part III.