Anyone doing LetsEncrypt with round robin DNS?

Comments

• arirang

  April 8

  I would only run Certbot on the front-end and use an internal private certificate for back-end communication, making sure CA verification is enforced. Make sure all external traffic is routed through the front-end.

  Thanked by (1)WSS
• AnthonySmith ModeratorHosting ProviderOGSenpai

  12:00AM

  My colleague came up with the method for TierHive, I will double check it tomorrow and give an overview probably works similar ISH to what you want or might give you an idea.

  Thanked by (1)WSS

  TierHive - Hourly VPS - NAT Native - /24 per customer - DE, UK, SG, CA, USA, FR
  FREE tokens on sign up, try before you buy. | Join us on Reddit
• jarland OGServices Provider

  12:34AM

  Might be overkill but I’ve been eyeballing it: https://github.com/certimate-go/certimate

  Finally reached the point in my life where I love a good API that I can automate from.

  Thanked by (1)WSS

  Do everything as though everyone you’ll ever know is watching.
• tetech OG

  9:16AM edited 9:18AM

  I have a separate container that handles the renewal process via DNS challenge, and rsyncs over wireguard to the haproxy frontends. From haproxy frontend to backend is a private certificate.

  Thanked by (4)Chievo skorous WSS quicksilver03
• AnthonySmith ModeratorHosting ProviderOGSenpai

  12:15PM

  From my colleague: for round robin, or in our case, we call it global distribution on the panel front end.

  1. Customer points all their DNS A records at every HAProxy external IP or the ones they want.
  2. Customer requests SSL through the panel
  3. The panel runs acme.sh on the designated ACME primary node only (HAProxy on our side is set up so 1 is always the master)
  4. acme.sh creates the challenge token file locally on the primary HAProxy server
  5. Let's Encrypt makes its HTTP validation request it could hit any of the HAProxy servers with A record pointed to them.
  6. Every non-primary node has its acme_backend in HAProxy configured to proxy like /.well-known/acme-challenge/ requests to the primary's internal IP which is reachable/redirected over the NAT Mesh.
  7. So regardless of which node Let's Encrypt hits, the challenge request is transparently forwarded to the primary where the token exists.
  8. Validation succeeds, certificate is issued on the primary HAProxy
  9. The panel then writes the PEM file directly to every other HAProxy node via SSH/SCP
  10. HAProxy is reloaded (Not restart) on each node to pick up the new certificate and then any entry point has the ssl.

  Old way on launch was using lsyncd, which is near instant replication (speed of latency) instead of rsync, but the above actually made it less fragile, on smaller scale or internal only not multi user, lsyncd might be a better option.

  Maybe that helps or gives an idea.

  Thanked by (3)yoursunny flips WSS

  TierHive - Hourly VPS - NAT Native - /24 per customer - DE, UK, SG, CA, USA, FR
  FREE tokens on sign up, try before you buy. | Join us on Reddit
• ralf

  1:52PM edited 1:56PM

  @WSS said:
  Hi Folks.

  I've got this zany idea of getting a backup service running on a NAT box behind HAProxy as well as the front-end with it's own dedicated IPs.

  I don't really want to rely on rsync, but I probably will have to to ensure data propagation properly- however, I don't really want to do that for SSL certificates for various reasons. I'm thinking possibly DNS challenge, or being really, really lazy and just letting either box submit it's certbot requests and waiting for the luck of the draw renew them every other day or so.

  Thoughts? Implementations on your side?

  It's not quite exactly what you're asking, but what I'm doing that seems to work well for me is I have certbot running on a VM on one of my dedis. All of my various web endpoints forward any /.well-known to the certbot VM (except for /.well-known/assetlinks.json which is used for IOS apps). Then certbot can do regular challenges, and it doesn't really matter which webserver handles the request.

  I originally had an ssh / rsync solution, but migrated towards having a .ssh/authorized_keys that just allows running a script that backs up the old certificates on that host, removes them and unpacks the tar file that's piped in, before restarting haproxy. I'm gradually shifting that the other way, so haproxy instances ssh into the certbot which just pipes them the correct tar file based on their SSH key because I want the haproxy VMs to decide themselves when to restart haproxy.

  For wildcard domains, it was more of a PITA and I do that on a different VM with a much newer certbot so that I can use --dns-rfc2136 and let it talk to my primary DNS server to allow it to update the DNS records. There was a gotcha there in that it's very easy for the automatically updated zones to conflict with manually edited ones, so in the end, I put _acme-challenge.example.com into its own zone file, and replicate that across all secondary DNS servers using regular DNS configs. That way you don't have to keep messing around with rndc freeze and rndc thaw which never seemed to work properly anyway. I also had to explicitly add a load of apparmor rules to give named rw access to every zone file and the corresponding .jnl file.

  I'd say setting certbot up to use DNS rather than forwarded HTTP felt like it was significantly more hassle than it was worth, but necessary for wildcard domains.
• quicksilver03 OG

  4:00PM

  The strategy I've been using at the latest 2 jobs and for my personal infrastructure is to decouple certificate generation from certificate distribution: I have acme.sh or an equivalent tool generate as many certificates I need using DNS challenges, then use a script to push all the files to a S3 bucket.
  The various nodes running nginx, HAProxy, etc. have a systemd timer that downloads the certificates from the S3 bucket and if there are any changes reloads the services.

  The advantage for me is having only one place where the certificate generation happens, and have a very simple certificate distribution mechanism. I also have SSL expiration monitors for the various endpoints to detect failed renewals.