In transit data security

All of the Fnordly web properties (this wiki, webmail, OpenStack API endpoints, etc) are hosted behind HAproxy daemon(s) running on the Internet facing firewall machines. While it would be nice to say that things are all covered by HAproxy doing TLS termination for us, defense in depth principles demand that all traffic that can be encrypted be encrypted. As such, traffic between the HAproxy endpoints and the internal web services is encrypted too, and web service identities are established with Let's Encrypted x509 certificates.

I should put a pretty picture here to make understanding a bit easier, but my graphical skills are extremely limited. As such, text will have to suffice for now.

Vision Statement

All conceivably TLSed traffic should be TLSed in transit and authenticated by valid (not expired) Let's Encrypt certificates. No ongoing manual certificate management should be needed. And private key + certificate reloading handled automatically as well. Tin foil hat on!

Inside network TLS encrypted services list

• All the HTTP things
  • Static web pages
  • Mediawiki content
  • Ceph rados gateway S3 and Swift services
  • Webmail
  • OpenStack API endpoints
  • Probably a few others
• IMAP
• SMTP
• probably missing an item or two here

Tools to be used

Lots of people like to hate on certbot, and probably for really good reasons. I have been successfully using it for a number of years, though. And intend to, for now, continue doing so. Apache HTTPD for the static web pages, MediaWiki, Roundcube (webmail), and other PHP applications. Postfix for SMTP. Dovecot for IMAP. uWSGI for the OpenStack API endpoints. Ceph radosgw for S3.

Let's Encrypt domain name ownership validation with be done by DNS-01 challenges. This allows a certificate requestor that is not directly accessible by Let's Encrypt's ACME validation servers to still request a certificate. Additionally, it allows for the issuance of wildcard certificates.

DNS contortions

As mentioned above, HAproxy on the internet facing firewalls will be doing the TLS termination for web (and other) clients. And it then, in turn, contacts the network-internal servers (be they VMs or bare metal) to actually get data for clients. The DNS-01 ACME challenge method requires that Let's Encrypt's servers be able to check a publicly visible DNS record to prove ownership. So for that to happen, the internal machine contacts an ACME server, gets a value to publish in the DNS, sends an RFC 2136 DNS update to the zone primary server, waits some time for it to be distributed to the secondaries, then asks the ACME server to check. If the ACME server can see the expected value in the external DNS, then it should issue a certificate.

But, I don't really want to publish the internal DNS zones to the world at large. At the same time, HAproxy will be validating certificates based on domain names. In order to minimize internal DNS information leakage, I have settled on creating a svc.fnord.greeley.co.us zone which exists in both the internal and external DNS views. The ACME requestor systems have credentials allowing updates to this zone. The external version of this zone is configured as an in-view zone in the BIND configuration so there is only one copy in the named process's memory. Updates to the interval view are seen immediately in the external view. And DNS NOTIFYs are sent to the secondary name servers as zone data updates are made.

Remember that "value to publish" two paragraphs up? It is put into a new DNS record that prefixes the requesting server's DNS name with _acme-challenge.. And, important to know... DNS zones are arranged in a hierarchy, but the records inside a zone are not. So inside a zone like svc.fnord.greeley.co.us, server-0.svc.fnord.greeley.co.us and _acme-challenge.server-0.svc.fnord.greeley.co.us exist independently. The latter does not depend on the former. This allows us to make server-0.svc.fnord.greeley.co.us a CNAME for server-0.internal.fnord.greeley.co.us and let the _acme-challenge.server-0.svc.fnord.greeley.co.us be visible to the outside world. (If a DNS label has a CNAME value attached, that label is not allowed to have any other vales attached.) As such, HAproxy (which can see the internal.fnord.greeley.co.us zone) can find the IP address (and other) information for server-0.svc.fnord.greeley.co.us by follwing the CNAME. But DNS clients outside the network cannot. Neat, right?

certbot installation and configuration

Packages are installed and configured by Salt states here. Salt minions that will be requesting certificates have the letsencrypt-certificate-requestor role pillar. And they also have a letsencrypt pillar listing domains (and parameters for them) for which they are requesting certificates.

As part of the package installation Salt state, a Let's Encrypt account is also registered by the ACME client machine, unless an account is already found.

Certificate enrollment

This is also performed by a Salt state. If the Salt minion has the letsencrypt-certificate-requestor role assigned, the following happens:

1. The /etc/letsencrypt/private directory is created if missing and its permissions set to root, root, 0700.
2. For each domain listed in the minion's letsencrypt:domains pillar value:
   1. A credentials file is stored under /etc/letsencyrpt/private. This file has the DNS TSIG secret necessary to make updates to the zone.
   2. Unless certbot already knows there is a certificate for the domain, a certificate is requested using the DNS-01 ACME challenge method.

certbot hook scripts

certbot requests certificates from Let's Encrypt. And it generates private keys, too. For TLS secured communications to happen, an endpoint needs its private key, its own ("leaf") certificate, and any intermediate certificates for certificate authorities (CAs) between the leaf and a trusted root CA. Different applications handle these in their own ways. So we may need to frobnicate the certbot PEM files or the applications' configuration to suit.

hook script for Ceph radosgw

Nothing needed here. The Ceph radosgw configuration points at the Let's Encrypt live key (prevkey.pem) and leaf cert + intermediate cert (fullchain.pem) file. The radosgw configuration is set to re-load the key and certificates every 15 minutes. As such, cert renewals are handled without any additional steps needed.

Well, hopefully. This will be tested soon. The radosgw process is supposed to run as a non-privileged ceph user, which may not be able to read the private key and certificate chain files.

certbot and/or Let's Encrypt annoyances

1. Running certbot --help security suggests that any elliptic curve supported in TLS 1.3 (defined in RFC 8446) should work for certificate creation. However, the Let's Encrypt ACME servers only support secp256r1 and secp384r1. (Tested empirically 2026-01-24.)