I've got this zany idea of getting a backup service running on a NAT box behind HAProxy as well as the front-end with it's own dedicated IPs.

I don't really want to rely on rsync, but I probably will have to to ensure data propagation properly- however, I don't really want to do that for SSL certificates for various reasons. I'm thinking possibly DNS challenge, or being really, really lazy and just letting either box submit it's certbot requests and waiting for the luck of the draw renew them every other day or so.

Thoughts? Implementations on your side?

I wrote a script using bash to help renew lets encrypt ssl issued using on a minio instance. Minio is an s3 compatible storage. Basically, the ssl is issued via certbot but files need to be moved around. I found myself every now and then having my minio ssl expired and I didn't know. so i created this script to check daily when the hostname ssl expires and to renew it automatically. hope it helps someone with a similar need.

#!/bin/bash
PUSHOVER_API_TOKEN='xxxxx'
PUSHOVER_API_USER='xxxx'

HOSTNAME=$(hostname)


DEFAULT_PRIVATEKEY_LOCATION=/etc/letsencrypt/live/$HOSTNAME/privkey.pem
DEFAULT_CERTIFICATE_LOCATION=/etc/letsencrypt/live/$HOSTNAME/fullchain.pem

MINIO_USERNAME=minio-user
MINIO_PRIVATEKEY_LOCATION=/etc/minio/certs/private.key
MINIO_CERTIFICATE_LOCATION=/etc/minio/certs/public.crt


certexpiredays(){
    EXPIREDATE=$(openssl x509 -in $MINIO_CERTIFICATE_LOCATION -enddate -noout | sed "s/.*=\(.*\)/\1/")
    EXPIRE_DATE_IN_SECONDS=$(date -d "${EXPIREDATE}" +%s)
    NOW_IN_SECONDS=$(date -d now +%s)
    DATE_DIFF_IN_SECONDS=$(( (EXPIRE_DATE_IN_SECONDS - NOW_IN_SECONDS) / 86400 ))

    echo $DATE_DIFF_IN_SECONDS
}


notify(){
    STATUS=$1

    local TITLE="Renewal of Minio SSL"
    if [[  $STATUS -eq 0  ]]
    then
        local MESSAGE="SUCCESS! Minio SSL successfully replenished for https://$HOSTNAME:9000"
    else
        local MESSAGE="FAILED! Could not complete SSL renewal for minio. Status code: $STATUS for https://$HOSTNAME:9000"
    fi
    local SOUND=siren
    local TIME=$(date)
    local PRIORITY=0
    curl -s \
      --form-string "token=$PUSHOVER_API_TOKEN" \
      --form-string "user=$PUSHOVER_API_USER" \
      --form-string "title=$TITLE" \
      --form-string "message=$MESSAGE" \
      --form-string "timestamp=$TIME" \
      --form-string "sound=$SOUND" \
      --form-string "priority=$PRIORITY" \
      https://api.pushover.net/1/messages.json
}


#0.Check if renewal is needed in the first place

DAYS_TO_SSL_EXPIRY=$(certexpiredays)


if [[ (( $DAYS_TO_SSL_EXPIRY < 5 ))]]
then
    #1.Renew SSL
    echo "===Commencing SSL renewal...==="

    echo 2 | sudo certbot certonly --standalone -d $HOSTNAME

    if [ $? -eq 0 ]
    then
        echo "==>OK! SSL for minio has been requested successfully."
        echo "==>Copying SSL files...."

        #2. Copy SSL files to correct directory
        sudo cp -v $DEFAULT_PRIVATEKEY_LOCATION $MINIO_PRIVATEKEY_LOCATION
        sudo cp -v $DEFAULT_CERTIFICATE_LOCATION $MINIO_CERTIFICATE_LOCATION

        if [ $? -eq 0 ]
        then
            echo "==>OK! SSL files copied successfully..."

            #3. Change ownership of the SSLs
            sudo chown $MINIO_USERNAME:$MINIO_USERNAME $MINIO_PRIVATEKEY_LOCATION
            sudo chown $MINIO_USERNAME:$MINIO_USERNAME $MINIO_CERTIFICATE_LOCATION

            if [ $? -eq 0 ]
            then
                #4. Restart minio service
                sudo systemctl restart minio

                pgrep minio >/dev/null 2>&1
                STATE=$(echo $?)

                if [[  $STATE == 0  ]]
                then
                    echo "==>OK! Minio restarted successfully..."

                    echo "==>SUCCESS! SSL for minio replenished successfully."
                    notify 0
                else
                    echo "==>FAILED! Could not restart minio."
                    notify 1
                    exit
                fi
            fi
        else
            echo "==>FAILED! There was an error copying SSL files."
            notify 2
            exit
        fi

    else
        echo "==>FAILED! There was an error requesting for SSL for minio."
        notify 3
        exit
    fi
else
    echo "==>SSL for minio still valid for $DAYS_TO_SSL_EXPIRY days. Nothing to do. Bye"
    exit
fi

On September 30, there will be a change in how older browsers and
devices trust Let's Encrypt certificates, resulting in a minor decrease
in compatibility. If you run a typical website, you won't notice a
difference. Devices and browsers running up-to-date software will
continue working fine, and we've taken steps to make sure the vast
majority of older devices will too. If you run a large website, or need
to support less common software (particularly non-browser software),
you'll want to read about the details at:

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

In either case, no action is required from you. We're letting you know
so you can provide answers to any questions your site visitors may have.

Here is a sample hostname from one of your current Let's Encrypt
certificates:

Since 2015 we've served the world with 1.6 billion free certificates,
each one providing security and privacy to people on the Web. It's work
that's 100% funded by charitable donations since we are a nonprofit. If
your company is interested in sponsorship, please email
sponsor@letsencrypt.org. If you can make a donation, we ask that you
consider supporting our work today: https://letsencrypt.org/donate/
Thank you.

• The Let's Encrypt team