Rclone backup setup for my 'hit-by-a-truck' protocol

bikegremlin · October 29

When my spare backup HDD at work died (and the warranty is going to take about a month), I configured encrypted Rclone sync from my PC HDD (source of truth) to a Hetzner storage box.

Made a step-by-step tutorial for my best man - my "hit-by-a-truck" protocol for when (motor)cycling goes wrong.

Any corrections are more than welcome - and if it helps other folks, even better:

https://io.bikegremlin.com/38747/rclone-installation-and-configuration-step-by-step/

Rclone should work with many cloud storage providers - it is just that Hetzner Storage Box has been reliable for hosting server backups for me over the years, and I was already paying for it, so I used that.

vyas · October 29

Hmm.. interesting !

Windows made me.. reheat my coffee. But then scrolling down further... it tasted better

bikegremlin · October 29

@vyas said:
Hmm.. interesting !

Windows made me.. reheat my coffee. But then scrolling down further... it tasted better

LOL. Good one.

Serbia is a Windows country (there was even some official Microsoft partnership at the start of the century).
Practically everyone I know uses Windows only.

vyas · October 29

I hear you...
on a serious note, the encryption part: why do it/ steps needed... very useful, yes even in age of AI agents

havoc · October 29

After losing data last week I’ve been rethinking backups too

Thinking mixture of zfs send, git and borgbackup.

AuroraZero · October 29

Are you saying you want me to hit you with a truck?

tetech · October 29

@havoc said:
After losing data last week I’ve been rethinking backups too

Thinking mixture of zfs send, git and borgbackup.

I am using btrfs send to push encrypted snapshots to S3 (minio). I do monthly full and weekly/daily incremental, with the 'parent' as tags, so a restore script can fetch everything needed to restore to a certain date, and the built-in S3 lifecycle cleans them up.

I also push the monthly backups to borgbackup for longer-term backups. So maybe not dissimilar to you.

hornet · October 29

@bikegremlin said: step-by-step tutorial

Terrific article!

havoc · October 29

Slightly OT - my (75 y/o) dad phoned me today asking about a user friendly GUI backup tool on Windows. And I didn't have an answer.

Can only think of Duplicati and the internet is full of reports of data corruption & sundry fails on it.

Fritz · October 29

@havoc said:
Slightly OT - my (75 y/o) dad phoned me today asking about a user friendly GUI backup tool on Windows. And I didn't have an answer.

Can only think of Duplicati and the internet is full of reports of data corruption & sundry fails on it.

Free File Sync might be a good option.

@tetech

Share on how you do it please.

tetech · 4:00AM

@havoc said:
Slightly OT - my (75 y/o) dad phoned me today asking about a user friendly GUI backup tool on Windows. And I didn't have an answer.

A few different approaches I've used: one is Veeam to my own server; one is iDrive; one is to use Bvckup to sync to a Samba share and then backup using Linux tools from there.

I used to use CrashPlan in the old days until the prices became prohibitive. I never used Backblaze but I have family members that do, similarly for Cloudberry (now MSP360).

bikegremlin · 4:13AM

@havoc said:
Slightly OT - my (75 y/o) dad phoned me today asking about a user friendly GUI backup tool on Windows. And I didn't have an answer.

Can only think of Duplicati and the internet is full of reports of data corruption & sundry fails on it.

Macrium Reflect is GUI for whole system images. Subscription only now, unfotunately.

Acronis too - subscription only too.

FileZilla if he wants to only copy files to a storage (Hetzner Storage Box is not expensive).

tetech · 4:55AM

@Fritz said:
@tetech

Share on how you do it please.

Mmm which part? As background info, I containerize everything using LXC on btrfs. However, the principle applies more generally: you don't need to use containers or btrfs, the below could be easily modified to any other type of snapshot-able filesystem.

I've got an outer loop something like this:

mon=$(date +%m)
if [ -f /var/lib/lxc/.snapshots/.month ]
then
  mod=$(cat /var/lib/lxc/.snapshots/.month)
else
  mod=
fi
echo "${mon}" > /var/lib/lxc/.snapshots/.month
d=$(date +%Y%m%d)
dow=$(date +%u)
for i in ${subvols}
do
  echo "Processing container $i..."
  ...
done

For each container (i.e. within the loop), I maintain a monthly and weekly snapshot that incremental backups are relative to. So on the first of the month I delete the monthly+weekly snapshot and on Sundays I delete the weekly snapshot:

# If new month, clear out all snapshots - this forces a full backup
if [ "${mon}" != "${mod}" ]
then
  if [ -d "/var/lib/lxc/.snapshots/${i}-month" ]
  then
    echo "  Clearing monthly snapshot"
    btrfs property set -f -ts "/var/lib/lxc/.snapshots/${i}-month" ro false
    btrfs sub del "/var/lib/lxc/.snapshots/${i}-month"
  fi
  if [ -d "/var/lib/lxc/.snapshots/${i}-week" ]
  then
    echo "  Clearing weekly snapshot"
    btrfs property set -f -ts "/var/lib/lxc/.snapshots/${i}-week" ro false
    btrfs sub del "/var/lib/lxc/.snapshots/${i}-week"
  fi
fi

# If a new week, remove the weekly snapshot to force weekly backup
if [ "${dow}" == "7" ] && [ -d "/var/lib/lxc/.snapshots/${i}-week" ]
then
  echo "  Clearing weekly snapshot"
  btrfs property set -f -ts "/var/lib/lxc/.snapshots/${i}-week" ro false
  btrfs sub del "/var/lib/lxc/.snapshots/${i}-week"
fi

Then the backup logic for each container works by detecting what snapshots are missing. So something like:

# Create daily snapshot - this is what we'll work with
daysnap="/var/lib/lxc/.snapshots/${i}-${d}"
btrfs sub snap -r "/var/lib/lxc/${i}" "${daysnap}"
echo "  Created working snapshot"

# Check whether full backup needed
csnap="/var/lib/lxc/.snapshots/${i}-month"
if [ ! -d "${csnap}" ]
then
  # Do a full backup & save to object storage
  echo "  Performing full backup"
  btrfs sub snap -r "${daysnap}" "${csnap}"
  PUTFN="${i}-monthly-full-${d}.gz${GPGEXT}"
  echo "${PUTFN}" > "/var/lib/lxc/.snapshots/.${i}.monthlyfn"
  PUTLEN=$(push_snapshot "${csnap}" "${i}" "monthly" "${PUTFN}")
else
  # Already have a full backup, so do a daily/weekly incremental
  psnap="${csnap}"
  csnap="/var/lib/lxc/.snapshots/${i}-week"
  if [ ! -d "${csnap}" ]
  then
    # Create new incremental weekly from month, if needed
    echo "  Performing weekly incremental backup"
    btrfs sub snap -r "${daysnap}" "${csnap}"
    if [ ! -f "/var/lib/lxc/.snapshots/.${i}.monthlyfn" ]
    then
      echo "${i}-monthly-full-$(date +%Y%m)01.gz" > "/var/lib/lxc/.snapshots/.${i}.monthlyfn"
    fi
    PUTFN="${i}-weekly-incr-${d}.gz${GPGEXT}"
    echo "${PUTFN}" > "/var/lib/lxc/.snapshots/.${i}.weeklyfn"
    parentfn=$(cat "/var/lib/lxc/.snapshots/.${i}.monthlyfn")
    PUTLEN=$(push_snapshot "${csnap}" "${i}" "weekly" "${PUTFN}" "${psnap}" "${parentfn}")
  else
    # Otherwise, create incremental daily from week
    echo "  Performing daily incremental backup"
    psnap="${csnap}"
    csnap="${daysnap}"
    if [ ! -f "/var/lib/lxc/.snapshots/.${i}.weeklyfn" ]
    then
      lsun=$(date -d "-$(date +%w) days" +%Y%m%d)
      echo "${i}-weekly-incr-${lsun}.gz" > "/var/lib/lxc/.snapshots/.${i}.weeklyfn"
    fi
    PUTFN="${i}-daily-incr-${d}.gz${GPGEXT}"
    parentfn=$(cat "/var/lib/lxc/.snapshots/.${i}.weeklyfn")
    PUTLEN=$(push_snapshot "${csnap}" "${i}" "daily" "${PUTFN}" "${psnap}" "${parentfn}")
  fi
fi
# Clean up our working snapshot
echo "  Cleaning up working snapshot"
btrfs property set -f -ts "${daysnap}" ro false
btrfs sub del "${daysnap}"

The missing piece of the puzzle is the push_snapshot function which sends it to S3 (minio) using rclone:

function push_snapshot() {
  local SNAPSAVE="${1}"
  local CONTAINER="${2}"
  local PERIOD="${3}"
  local PUTFN="${4}"
  local REL=()
  local HDR=()

  if [[ "${PERIOD}" != "monthly" ]]
  then
    REL=(-p ${5})
    local PARENTFN=${6}
    HDR+=(--header-upload "x-amz-tagging: period=${PERIOD}&host=${hshort}&parent=${PARENTFN}")
    HDR+=(--header-upload "x-amz-meta-parent: ${PARENTFN}")
  else
    HDR+=(--header-upload "x-amz-tagging: period=${PERIOD}&host=${hshort}")
  fi

  local UUID=$(btrfs sub show "${SNAPSAVE}" | grep "UUID:" | head -1 | awk '{print $2}')
  HDR+=(--header-upload "x-amz-meta-btrfs: ${UUID}")
  rm -f /tmp/btrfs.send
  mkfifo /tmp/btrfs.send
  wc -c < /tmp/btrfs.send > /tmp/btrfs.len &

  timeout 8h btrfs send "${REL[@]}" "${SNAPSAVE}" | gzip -c | ${GPGCMD} | tee -p /tmp/btrfs.send | rclone rcat --s3-no-check-bucket --s3-no-head --s3-no-head-object "lxcbu:${BUCKET}/${CONTAINER}/${PUTFN}" "${HDR[@]}"

  sleep 1
  cat /tmp/btrfs.len
  rm /tmp/btrfs.send /tmp/btrfs.len
}

In this function, the headers are so that a restore script can fetch all the necessary parts to reassemble a point-in-time snapshot. Set GPGCMD to the encryption process. But the actual snapshot backup is basically one line (and it doesn't create a whole lot of temporary files, so does OK if there's limited/full disk).

I do a few extra things like cpu limiting, reading parameters from config files, and getting a "token" from my minio server to limit the number of concurrent uploads - I have several dozen hosts and if they all try to back up their containers simultaneously it kills the I/O of the minio server.

Is that what you wanted to know?

tetech · 5:08AM

How it looks in minio. The backup list:

The metadata which is used to reassemble the snapshot:

Everything is encrypted before reaching the server so the only thing "exposed" is the container name. So it should be OK for any S3 storage, but mine is on my own minio.

Rclone backup setup for my 'hit-by-a-truck' protocol

Comments