yoursunny.com is Served by Caddy

yoursunny · January 2021

This is a personal story on how I'm keeping a 15-year-old website alive.
Read along if you want to see a comparison among the various HTTP servers and TLS certificate solutions I used during the past 10 years.
This post is originally published on yoursunny.com blog https://yoursunny.com/t/2021/yoursunny-com-caddy/

The last rebuild of yoursunny.com was in Spring 2017, when I moved the whole website into git repositories.
It's been more than 3 years, and I think I should share an update on a few changes in the stack that serves this website.

History of HTTP Servers Behind yoursunny.com

Since 2011, my HTTP server of choice was lighttpd.
Then, I have PHP running in FastCGI mode to serve the dynamic pages.
It works, but I don't really like the lighttpd's script-like configuration structure.
Moreover, there were suspected memory leaks in my setup, so that I had to use a cron job to restart the HTTP server weekly.

I keep hearing good words about nginx, as well as the benefits of running PHP in FPM mode.
In 2013, I made the switch to nginx and PHP-FPM.
The declarative configuration of nginx is easy to understand and makes sense to me.

HTTPS came to yoursunny.com at the end of 2015.
Like many other websites, the TLS certificates were issued by Let's Encrypt, and requested through certbot command line client.
One problem is that, my VPS at the time had only 64MB memory, and certbot would not work in such a small amount of memory.
I had to request the TLS certificate on my laptop, and then upload it to the VPS.

I bought a larger VPS with 1GB memory in 2016.
With ample memory, certbot was able automatically obtain the certificate every other month.
However, I soon found that although certbot wrote the new certificates to the filesystem, nginx would continue to use the old certificates until it is restarted.
Thus, every time I receive a Certificate Transparency Notification email, I had to SSH into the server and restart nginx.
I'm sure there are ways to automate this, but I never tried to figure out.

In November 2020, I had a brief interaction with acme.sh, an ACME protocol client written in Unix Shell language.
It's easy to understand, consumes minimal resources, and can be configured to restart nginx very easily.
acme.sh quickly replaced certbot in all my setups, until I found an easier way.

Caddy HTTPS Server

nginx is fast, and acme.sh handles all my certificate needs.
I followed the advice on Mozilla SSL Configuration Generator and received an "A" rating on SSL Server Test.
Things have been working well so far.

A few years later, I noticed that my green "A" rating turned into a yellow "B" rating, with a warning message:

This server supports TLS 1.0 and TLS 1.1. Grade capped to B.

What happened?

It turns out that, HTTPS is more than a padlock: as the Internet evolves, the security recommendations are also evolving.
An nginx configuration written in 2016, adapted to the browsers and mobile devices at that time, is no longer secure in 2020, because over the years, protocol vulnerabilities are discovered and attacks are becoming more sophisticated.
If I want to keep the "A" rating, I have to keep the nginx configuration up to date with the current security recommendations.

Then I discovered an easier method: there's a new HTTP server called Caddy.
Caddy is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go.
Caddy automatically obtains and renews TLS certificates, uses a hardened TLS stack powered by the Go standard library, and has a secure-by-default TLS configuration complete with OCSP stapling.

Instead of:

nginx
acme.sh or certbot
periodically restart nginx
annual nginx configuration check-up

I just need:

Caddy 2

Caddy + nginx + PHP-FPM

If I want to run a simple HTTPS server serving static files, I just need:

Caddy 2

In reality, yoursunny.com is a 15-year-old website with hundreds of pages.
Apart from HTTPS, I also need:

PHP script to render Markdown into HTML
PHP script to retrieve my Twitter updates
PHP script to display a list of blog articles on the frontpage
redirect rules to keep the query strings working in /study subsite
redirect rules to keep old links under /work/ directory working

All these have been written in the nginx configuration.
Caddy is powerful and robust, but I'm not ready to rewrite all these into a Caddyfile.
Moreover, a critical feature is missing in Caddy: there isn't a caching layer.

How I designed my PHP scripts is that, they depend on the HTTP server to support caching.
For example, I have a PHP script that retrieves my recent tweets and then render them into an HTML snippet.
Every execution of this script involves a network request and a Twig template execution, which would consume quite some server resources.
If I was working with a shared hosting account, the script would have a temporary file mechanism such that it only performs the expensive operations every few minutes.
However, since I have been able to control the HTTP server, I shifted the caching responsibility to nginx, so that the PHP script itself would be simpler.

To solve these problems, I decided to keep using nginx for the website logic.
Therefore, the current setup is:

Caddy:
- Redirect plain HTTP (port 80) to HTTPS (port 443).
- Terminate TLS.
- Accept HTTP/1 and HTTP/2 requests on port 443.
- Forward all requests to nginx over a Unix socket.
- HTTP/3 is disabled at the moment, because my current hosting provider has a DDoS protection firewall that rate-limits IPv4 UDP traffic.
nginx:
- Process all HTTP/1.1 requests.
- Handle redirect rules.
- Forward PHP requests to PHP-FPM over another Unix socket.
- Cache FastCGI responses from PHP scripts.
- Static files are handled by nginx rather than Caddy, because some of my redirect rules are written as 404 handlers.
PHP-FPM:
- Execute all my scripts.

Security Headers

Having Caddy handle TLS termination, yoursunny.com is back to the green "A" rating on SSL Server Test.
Then I found a new website security rating system: Security Headers.
It verifies and makes recommendations on a few security-related HTTP response headers from my server.

Starting from a big red "F", I have been reading the guidelines and making changes to the Caddyfile as I begin to understand what they are about.
As of this writing, I'm receiving a yellow "B" rating on Security Headers:

✔️ Referrer-Policy
✔️ Strict-Transport-Security
✔️ X-Content-Type-Options
✔️ X-Frame-Options
❌ Content-Security-Policy
❌ Permissions-Policy

Not bad, if you consider that Google has a "D" and Cloudflare has a "C".

Conclusion

This article goes behind the scenes of my website yoursunny.com.
It describes my evolving choice of HTTP servers and TLS solutions, and the current setup that combines Caddy and nginx.
Caddy and nginx configuration files behind this website, along with all other source code, are published in the yoursunny-website git repository.

Appendix

List of providers I used in recent years:

LoliHosting - deadpooled
VPSCheap - too much downtime
W00tH0sting - deadpooled
SecureDragon - used to be nice
@SMARTHOST - very stable and fast
SpartanHost - good speeds to China, but DDoS protection rate-limits UDP so I can't have HTTP/3 and DDoS protection at the same time

dahartigan · January 2021

How many push-ups did you neglect to do because you used that time writing up that report?

Freek · January 2021

Nice write-up. I too recently switched to Caddy for one of my projects. Previously I was using nginx and the letsencrypt nginx proxy companion for SSL certificates (both Docker images, since I'm a Docker guy) but I found it unnecessary complicated and bloated. Caddy eliminates that. It took me about 20-30 minutes to understand their syntax and setup. Issue is that the syntax from Caddy 2 is different from Caddy 1 in most cases, which can be annoying if stumbling upon old/outdated documentation.

lowendmeow · January 2021

I've heard great things about caddy and have wanted to try too especially for its reverse proxy ability. thanks for the writeup! @yoursunny btw your site seems to be down though..

ehab · January 2021

caddy had some hiccups and didn't get time to learn the new syntax when 2 came so i stick with

https://traefik.io/traefik/

yoursunny · January 2021

@dahartigan said:
How many push-ups did you neglect to do because you used that time writing up that report?

22

@Freek said:
Issue is that the syntax from Caddy 2 is different from Caddy 1 in most cases, which can be annoying if stumbling upon old/outdated documentation.

I only read Caddyfile reference.

@lowendmeow said:
@yoursunny btw your site seems to be down though..

PHP-FPM crashed. First time this year.
It doesn't affect the blog that is a static subsite.

hey · January 2021

Instead of:

use --renew-hook

ouvoun · January 2021

I friggin love Caddy. Best tool I’ve used in a long time. Makes it beautifully easy to handle reverse proxies.

CamoYoshi · January 2021

Thanks for the really interesting writeup! A bit more recently certbot got nginx support added so you don't have to restart it for it to start using the certificates after renewal, so it's fully automated. I think you just have to pass --nginx when running the certificate request.

dahartigan · January 2021

I'm going to throw haproxy into this thread, it's an awesome lightweight reverse proxy and so far there's nothing I haven't been able to do with it.

hey · January 2021

@dahartigan said: I'm going to throw haproxy into this thread, it's an awesome lightweight reverse proxy and so far there's nothing I haven't been able to do with it.

don't forget to put multiple Nginx behind the haproxy

yoursunny · January 2021

OGF has been bashing my /phpinfo.php. Thus, I just launched an improved version of phpinfo().
Check it out: https://yoursunny.com/phpinfo.php

@CamoYoshi said:
A bit more recently certbot got nginx support added so you don't have to restart it for it to start using the certificates after renewal, so it's fully automated.

Yes, I heard, but I never tried it because I fear it would mess up my handwritten config files.

@dahartigan said:
I'm going to throw haproxy into this thread, it's an awesome lightweight reverse proxy and so far there's nothing I haven't been able to do with it.

I hear haproxy is used by most NAT providers so that users can put websites on port 80 of the shared IPv4.
However, I wouldn't need it when I have dedicated IPv4.

ehab · January 2021

@yoursunny said:
I hear haproxy is used by most NAT providers so that users can put websites on port 80 of the shared IPv4.
However, I wouldn't need it when I have dedicated IPv4.

haproxy is the master of all proxies. i mean it.

havoc · January 2021

hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

CamoYoshi · January 2021

@yoursunny said:

@CamoYoshi said:
A bit more recently certbot got nginx support added so you don't have to restart it for it to start using the certificates after renewal, so it's fully automated.

Yes, I heard, but I never tried it because I fear it would mess up my handwritten config files.

It actually is pretty smart about it; it only ever adds snippets to your config (it also gives you a choice to do it manually and just get the certificates), and said snippets are commented "Managed by Certbot" so you can tell what it did. It also, for second-level domains, gives you the option to force redirection to non-www, as well as enforce HTTPS redirection. It's really well thought out and I am extremely happy so far!

akhfa · January 2021

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

You will love to use traefik when using it as reverse proxy in docker

ouvoun · January 2021

@akhfa said:

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

You will love to use traefik when using it as reverse proxy in docker

Man I tried traefik 1 for a bit. It was buggy. Then I tried traefik 2. In both cases the configuration was so much more verbose and full of magic and/or arcane configuration when compared to caddy.

akhfa · January 2021

@ouvoun said:

@akhfa said:

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

You will love to use traefik when using it as reverse proxy in docker

Man I tried traefik 1 for a bit. It was buggy. Then I tried traefik 2. In both cases the configuration was so much more verbose and full of magic and/or arcane configuration when compared to caddy.

I agree that it has learning curve. But once you get a grip about how it works, based on my experience, it feels most "native way" to be deployed on "cloud environments"

For example, when I tried to use nginx to reverse proxy wordpress in docker, when we upgrade the WordPress by changing its image tag, the WordPress container will be recreated and will have new IP. In this case, nginx will lost connection to the WordPress and usually I restart the nginx container so that it can detect wordpress's new IP

You need to use work around like as https://github.com/nginx-proxy/nginx-proxy to solve above problem

For caddy, I'm not sure because I don't use it yet, but a little googling give me https://github.com/lucaslorentz/caddy-docker-proxy

That is what I mean with "native way". Traefik doesn't need workaround to achieve "dynamic reverse proxy" capabilities
But of course every tech has its own usage, for example traefik is designed to be reverse proxy and cannot serve files.

Wolveix · January 2021

@akhfa said: For example, when I tried to use nginx to reverse proxy wordpress in docker, when we upgrade the WordPress by changing its image tag, the WordPress container will be recreated and will have new IP. In this case, nginx will lost connection to the WordPress and usually I restart the nginx container so that it can detect wordpress's new IP

If a container's IP is ever important, you should be setting it as static anyway. So unfortunately, this use-case doesn't really provide any real reason to use Traefik over Caddy. It's as simple as setting a container's IP (which can be provided within the original run command or compose file), running Caddy either as a standalone binary or as a container itself, and writing in Caddy's incredibly simplistic configuration syntax. Not to mention that Caddy is a full-fledged web server rather than being just a reverse proxy, which can be useful if you need to do complex things with specific services running within or outside of Docker containers.

vyas · January 2021

@yoursunny said:

@dahartigan said:
How many push-ups did you neglect to do because you used that time writing up that report?

22

Correct answer is 42

Love the new phpinfo page

ouvoun · January 2021

@akhfa said:

@ouvoun said:

@akhfa said:

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

You will love to use traefik when using it as reverse proxy in docker

Man I tried traefik 1 for a bit. It was buggy. Then I tried traefik 2. In both cases the configuration was so much more verbose and full of magic and/or arcane configuration when compared to caddy.

I agree that it has learning curve. But once you get a grip about how it works, based on my experience, it feels most "native way" to be deployed on "cloud environments"

For example, when I tried to use nginx to reverse proxy wordpress in docker, when we upgrade the WordPress by changing its image tag, the WordPress container will be recreated and will have new IP. In this case, nginx will lost connection to the WordPress and usually I restart the nginx container so that it can detect wordpress's new IP

You need to use work around like as https://github.com/nginx-proxy/nginx-proxy to solve above problem

For caddy, I'm not sure because I don't use it yet, but a little googling give me https://github.com/lucaslorentz/caddy-docker-proxy

That is what I mean with "native way". Traefik doesn't need workaround to achieve "dynamic reverse proxy" capabilities
But of course every tech has its own usage, for example traefik is designed to be reverse proxy and cannot serve files.

Yeah I understand where you’re coming from, but it was in fact the dynamic nature of traefik that made me dislike it. Because it would randomly break.

For reference, here’s a fully functional reverse proxy in Caddy with automatic TLS certificates and all that jazz:

service.{$DOMAIN} {
  reverse_proxy localhost:8080
}

That’s it. And here’s a static site with automatic HTTPS, of course:

{$DOMAIN} {
  root * /www
  file_server
}

Not bad!

Freek · January 2021

@ouvoun said:

@akhfa said:

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

You will love to use traefik when using it as reverse proxy in docker

Man I tried traefik 1 for a bit. It was buggy. Then I tried traefik 2. In both cases the configuration was so much more verbose and full of magic and/or arcane configuration when compared to caddy.

This sums up my experience with Traefik as well.

havoc · January 2021

@ouvoun said:

@akhfa said:

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

You will love to use traefik when using it as reverse proxy in docker

Man I tried traefik 1 for a bit. It was buggy. Then I tried traefik 2. In both cases the configuration was so much more verbose and full of magic and/or arcane configuration when compared to caddy.

Traefik works well in a docker context - said magic is geared towards that. Outside of that less so

elliotc · January 2021

@havoc said:
hmm...I've used caddy for reverse proxy in a docker type scenario. Didn't know it could web serve too.

Interesting...have been doing that via nginx till now.

Thanks OP

I was using caddy for docker too, but they launch caddy2.0 and change the config format. It turned out I returned to nginx.

Wolveix · January 2021

@elliotc said: I was using caddy for docker too, but they launch caddy2.0 and change the config format. It turned out I returned to nginx.

You can still use Caddyfile syntax which has barely changed. Of course, you can also continue to use Caddy v1.

yoursunny.com is Served by Caddy

History of HTTP Servers Behind yoursunny.com

Caddy HTTPS Server

Caddy + nginx + PHP-FPM

Security Headers

Conclusion

Appendix

Comments