I got a cheap storages boxes like 1TB NAT for 10$/y and I had to put it into use.
However not fully trusting them with raw vm images, so PGP will do.

The backups are pgp encrypted before they are getting pulled by the storage server.
The proxmox host has never any access to the backup server and the backup server never has any access to the vm images since they are encrypted.

Whatever gets compromised, your should still be safe, that's the idea.
Downside is obviously, disk usage, for every backup we do in Proxmox, we have to do another encrypted copy and wait for the backup server to pull it before we can clear the disk space.

This can easily be done by using the Proxmox hooks, so you can configure your backup schedule like you normaly would do, only the STOREID has to match.
The only thing you have to tweak is, the backup server when it has to pull the backups.

1. Install the hook.

I do daily/weekly offside backups and keep them for 4 weeks.
Example for weekly, script can be modded though.

https://pastebin.com/raw/31jMWKTz
Wouldn't format properly...

Put the script to /usr/local/bin/vzdump-hook.sh and make it executable.
Don't forget to create the user "weekly" and the folder structure (/mnt/weekly/dump/ or whatever folder you want to use)
You also should install and ssh key for the backup server to be able to login into the user "weekly".

In my case, Proxmox creates usuable backups to /mnt/weekly/dump and the encrypted ones are put to /home/weekly/backups for pulling.

2. Edit /etc/vzdump.conf

Replace
#script: FILENAME
with
script: /usr/local/bin/vzdump-hook.sh

3. Either generate or import a existing pgp key

gpg --gen-key

or

gpg --import mahkey

Make SURE you backup this key.

4. Make a test backup and check if the encrypted backup is there.
There should be zero errors in the proxmox backup log.

If you do, you might have to trust that pgp key.
gpg --edit-key mahkey
and type "trust"

5. Setup a cronjob on the remote storage server

5 5 * * * /home/weekly/backups.sh

You might have to adjust the time.
rsync will do.

#!/bin/bash
set -e
if pgrep -fl backup.sh &>/dev/null; then
        rsync -Pav -e "ssh -i weekly" weekly@mahserverip:/home/weekly/backups/* /home/weekly/backups/
        find /home/weekly/backups/* -mtime +30 -exec rm {} +
fi

Don't forgot to make it executable and do a testrun.

6. Profit.

Thanks for reading my TED TALK.

GPG Encrypted data network.

Krypton (from Ancient Greek: κρυπτός, romanized: kryptos 'the hidden one').

Download: https://krypton.monster/krypton.zip

VirusTotal: https://www.virustotal.com/gui/url/7a12cd820fe15d8a4520038bde556cc78efdab3595f2270c71ff25bd15802de5/detection

Requirements: gpg, gpg-agent, curl

Usage: ./krypton "PRiVATE_KEY" "DATA_PASSWORD" "DATA"

Mirror the data: wget -m -np -c -R "index.html*" "https://krypton.monster/data/"

Verify: Every submitted GPG data will end having filename as SHA256 hash.
We use checksums, so you can always verify that the file was untouched - https://krypton.monster/data/0xF6FF5ADAB5180A3D10FFB611F2CADDD0A2B0922BDE398AD186E946480BEC3943/checksums.