VPS maintenance frameworks?

wankel

Hi all,

I've managed to keep most of my systems somewhat up-to-date by running a hodgepodge of bash scripts that would blindly try to apt update and upgrade each of them.

It has somewhat worked, with some manual intervention here and there, but 'top notch' is not the stamp I'd put on the works.

With the xz business going on, I see some more updates coming our way, so this could be the moment to improve the upgrading process overall.

My machines mostly run Debian (11 and 12, a single one perhaps 10), with a few Alpines.

What do you use or suggest to centralize maintenance of at least the Debian machines, or LInux servers in general?

somik

From a central server that has ssh access to all of your other servers:

#!/bin/bash

ssh -i ~/.ssh/id_rsa_10 -p 8022 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_11 -p 8122 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_12 -p 8222 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_13 -p 8322 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_14 -p 8422 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_15 -p 8522 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_16 -p 8622 [email protected] "sudo apt update && sudo apt upgrade" &

Note the different ssh keys (you can use the same one if you want) and use of non standard ssh port, different for each server.

You can also use a bastion server, but you have to figure it out yourself on how to set it up as I have yet to set one up for myself.

VirtFusion

@somik said:
From a central server that has ssh access to all of your other servers:

#!/bin/bash

ssh -i ~/.ssh/id_rsa_10 -p 8022 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_11 -p 8122 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_12 -p 8222 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_13 -p 8322 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_14 -p 8422 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_15 -p 8522 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_16 -p 8622 [email protected] "sudo apt update && sudo apt upgrade" &

Note the different ssh keys (you can use the same one if you want) and use of non standard ssh port, different for each server.

You can also use a bastion server, but you have to figure it out yourself on how to set it up as I have yet to set one up for myself.

How do you deal with services that need to be restarted? It looks like you are blindly updating all packages.

wankel

@VirtFusion said: It looks like you are blindly updating all packages.

Indeed, at the moment that is what I do

@somik said:

From a central server that has ssh access to all of your other servers:
...
Note the different ssh keys (...)

That's a nice touch... though, I'm not quite sure in which case it helps me more than that it complicates matters (an attacker that has access to two of my machines can see that it has the same public key and infer that the corresponding private key might have access to more, but once someone got hold of the private key, they'd have access to the 'central' server and thus to the separate keys as well.

I have a list of servers that I feed to a somewhat similar, slightly more dynamic, script, I'd hoped someone already had taken the trouble to create something to help managing a set of machines. Searching online gives Netbox (network management), Kubernetes (container management), Cobbler (server provisioning) as solutions to related, but slightly different problems.

/edit/ I don't really have an idea of the name of the class of software I'd be looking for, any hints?

somik

@VirtFusion said:
How do you deal with services that need to be restarted? It looks like you are blindly updating all packages.

I am giving an example here. You can replace the sudo apt update && sudo apt upgrade with your preferred command or set of command. You can even specify a specific bash script on your server to run instead of this.

ssh -i ~/.ssh/id_rsa_10 -p 8022 [email protected] "sudo sh /home/wankel/start_update.sh" &

You can also use unattended updates: https://www.baeldung.com/linux/automatic-updates-status-configuration

@wankel said:
That's a nice touch... though, I'm not quite sure in which case it helps me more than that it complicates matters (an attacker that has access to two of my machines can see that it has the same public key and infer that the corresponding private key might have access to more, but once someone got hold of the private key, they'd have access to the 'central' server and thus to the separate keys as well.

Sorry, you lost me there... The server that runs and manages the updates needs access to all of your other servers. All the methods you mention bellow uses the same method. Moreover, your servers can have different public/private keys. I believe I mentioned that clearly in my post.

Note the different ssh keys

So you have 1 pair of public/private key for each server. Your central master server holds all private keys while your slave servers (the ones that gets updated) gets 1 public key each. Usually your central server is not accessible over the internet (to prevent misuse).

I have a list of servers that I feed to a somewhat similar, slightly more dynamic, script, I'd hoped someone already had taken the trouble to create something to help managing a set of machines. Searching online gives Netbox (network management), Kubernetes (container management), Cobbler (server provisioning) as solutions to related, but slightly different problems.

Since I do not know what you need to do or run, i cannot tell you want to do for your specific use case. If I use one of my servers as an example, I usually run:

# weekly
ssh -p xxxx docker.vm "sudo sh /home/somik/update_docker.sh" &

# monthly
ssh -p xxxx docker.vm "sudo sh /home/somik/update_and_restart.sh" &

wankel

Thanks for taking the time to elaborate!

@somik said: I mentioned [unique keys per server] clearly in my post.

Yes, you did! I wondered in which case it would increase security or usability.

@somik said: one of my servers as an example
[remotely executed script]

I now have a bunch of scripts that ssh into $1 with a maintenance user, some of which execute $2. In that directory I touch new.domain.tld when I got a new VPS, so that bash helps me completing $1 when calling a script.

That is OK for one-off commands for a specific server, but having bash accept an array as input comes (in case of my limited bash-fu) with some headaches. Meaning I can not drop a whole bunch of domains in the script as separate variables (to elaborate on that: I intended to have a management.domain.tld for each server, so I can call a script as ./serverupdate.sh management.*, by having each server referenced by a management.-domain as empty file in that directory).

somik

You can also make use of bash arrays to pass a list of hosts:

hosts=("management.domain1.tld" "test.example.com" "forum.example.com")

And us them in your bash in a array:

for host in "${hosts[@]}"
do
    sh ./serverupdate.sh $host
done

terrorgen

I use NixOS plus colmena.

IAmNix

Ansible.

It's a sysadmin framework that abstracts away a lot of the OS specifics, but still lets you write simple code that can either just run pure bash, or do simple/advanced sysadmin things like user management, configure services etc.

You run them from your local computer, and they SSH in to your hosts. No dependencies expect python.

Here's an example that does what you need. You need to configure your inventory file, then run the below playbook with ansible-playbook -i inventory ./upgrade_all.yml

https://www.jeffgeerling.com/blog/2022/ansible-playbook-upgrade-ubuntudebian-servers-and-reboot-if-needed

wankel

Thanks a lot guys!

@somik said: You can also make use of bash arrays to pass a list of hosts:

I'm glad to see that my script-based approach is not totally outdated. Thanks for the array tip :-)

@terrorgen said: I use NixOS plus colmena.

I heard good things about NixOS over the years, so it definitely is on the list to try out once.

@IAmNix said: Ansible (...) is a sysadmin framework

Ansible may just be the answer I have been looking for, thanks for the introduction!