VPS maintenance frameworks?

wankelwankel OG
in Help

Hi all,

I've managed to keep most of my systems somewhat up-to-date by running a hodgepodge of bash scripts that would blindly try to apt update and upgrade each of them.

It has somewhat worked, with some manual intervention here and there, but 'top notch' is not the stamp I'd put on the works.

With the xz business going on, I see some more updates coming our way, so this could be the moment to improve the upgrading process overall.

My machines mostly run Debian (11 and 12, a single one perhaps 10), with a few Alpines.

What do you use or suggest to centralize maintenance of at least the Debian machines, or LInux servers in general?

Comments

  • somiksomik OG
    edited April 1

    From a central server that has ssh access to all of your other servers:

    #!/bin/bash

ssh -i ~/.ssh/id_rsa_10 -p 8022 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_11 -p 8122 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_12 -p 8222 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_13 -p 8322 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_14 -p 8422 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_15 -p 8522 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_16 -p 8622 [email protected] "sudo apt update && sudo apt upgrade" &

    Note the different ssh keys (you can use the same one if you want) and use of non standard ssh port, different for each server.

    You can also use a bastion server, but you have to figure it out yourself on how to set it up as I have yet to set one up for myself.

    Thanked by (1)wankel

    Artificial intelligence is no match for our natural stupidity.

    Time flies like an arrow; fruit flies like a banana.

  • VirtFusionVirtFusion Services Provider

    @somik said:
    From a central server that has ssh access to all of your other servers:

    #!/bin/bash

ssh -i ~/.ssh/id_rsa_10 -p 8022 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_11 -p 8122 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_12 -p 8222 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_13 -p 8322 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_14 -p 8422 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_15 -p 8522 [email protected] "sudo apt update && sudo apt upgrade" &
ssh -i ~/.ssh/id_rsa_16 -p 8622 [email protected] "sudo apt update && sudo apt upgrade" &

    Note the different ssh keys (you can use the same one if you want) and use of non standard ssh port, different for each server.

    You can also use a bastion server, but you have to figure it out yourself on how to set it up as I have yet to set one up for myself.

    How do you deal with services that need to be restarted? It looks like you are blindly updating all packages.

    Thanked by (2)wankel skhron

    VirtFusion Affordable, Reliable virtualization management software for the hosting industry · Connect with us on Discord

  • wankelwankel OG
    edited April 1

    @VirtFusion said: It looks like you are blindly updating all packages.

    Indeed, at the moment that is what I do

    @somik said:

    From a central server that has ssh access to all of your other servers:
    ...
    Note the different ssh keys (...)

    That's a nice touch... though, I'm not quite sure in which case it helps me more than that it complicates matters (an attacker that has access to two of my machines can see that it has the same public key and infer that the corresponding private key might have access to more, but once someone got hold of the private key, they'd have access to the 'central' server and thus to the separate keys as well.

    I have a list of servers that I feed to a somewhat similar, slightly more dynamic, script, I'd hoped someone already had taken the trouble to create something to help managing a set of machines. Searching online gives Netbox (network management), Kubernetes (container management), Cobbler (server provisioning) as solutions to related, but slightly different problems.

    /edit/ I don't really have an idea of the name of the class of software I'd be looking for, any hints?

  • somiksomik OG

    @VirtFusion said:
    How do you deal with services that need to be restarted? It looks like you are blindly updating all packages.

    I am giving an example here. You can replace the sudo apt update && sudo apt upgrade with your preferred command or set of command. You can even specify a specific bash script on your server to run instead of this.

    ssh -i ~/.ssh/id_rsa_10 -p 8022 [email protected] "sudo sh /home/wankel/start_update.sh" &

    You can also use unattended updates: https://www.baeldung.com/linux/automatic-updates-status-configuration

    @wankel said:
    That's a nice touch... though, I'm not quite sure in which case it helps me more than that it complicates matters (an attacker that has access to two of my machines can see that it has the same public key and infer that the corresponding private key might have access to more, but once someone got hold of the private key, they'd have access to the 'central' server and thus to the separate keys as well.

    Sorry, you lost me there... The server that runs and manages the updates needs access to all of your other servers. All the methods you mention bellow uses the same method. Moreover, your servers can have different public/private keys. I believe I mentioned that clearly in my post.

    Note the different ssh keys

    So you have 1 pair of public/private key for each server. Your central master server holds all private keys while your slave servers (the ones that gets updated) gets 1 public key each. Usually your central server is not accessible over the internet (to prevent misuse).

    I have a list of servers that I feed to a somewhat similar, slightly more dynamic, script, I'd hoped someone already had taken the trouble to create something to help managing a set of machines. Searching online gives Netbox (network management), Kubernetes (container management), Cobbler (server provisioning) as solutions to related, but slightly different problems.

    Since I do not know what you need to do or run, i cannot tell you want to do for your specific use case. If I use one of my servers as an example, I usually run:

    # weekly
ssh -p xxxx docker.vm "sudo sh /home/somik/update_docker.sh" &

# monthly
ssh -p xxxx docker.vm "sudo sh /home/somik/update_and_restart.sh" &
    Thanked by (1)wankel

    Artificial intelligence is no match for our natural stupidity.

    Time flies like an arrow; fruit flies like a banana.

  • wankelwankel OG

    Thanks for taking the time to elaborate!

    @somik said: I mentioned [unique keys per server] clearly in my post.

    Yes, you did! I wondered in which case it would increase security or usability.

    @somik said: one of my servers as an example
    [remotely executed script]

    I now have a bunch of scripts that ssh into $1 with a maintenance user, some of which execute $2. In that directory I touch new.domain.tld when I got a new VPS, so that bash helps me completing $1 when calling a script.

    That is OK for one-off commands for a specific server, but having bash accept an array as input comes (in case of my limited bash-fu) with some headaches. Meaning I can not drop a whole bunch of domains in the script as separate variables (to elaborate on that: I intended to have a management.domain.tld for each server, so I can call a script as ./serverupdate.sh management.*, by having each server referenced by a management.-domain as empty file in that directory).

  • somiksomik OG

    You can also make use of bash arrays to pass a list of hosts:

    hosts=("management.domain1.tld" "test.example.com" "forum.example.com")

    And us them in your bash in a array:

    for host in "${hosts[@]}"
do
    sh ./serverupdate.sh $host
done
    Thanked by (1)wankel

    Artificial intelligence is no match for our natural stupidity.

    Time flies like an arrow; fruit flies like a banana.

  • terrorgenterrorgen OG

    I use NixOS plus colmena.

    Thanked by (1)wankel

    The all seeing eye sees everything...

  • IAmNixIAmNix

    Ansible.

    It's a sysadmin framework that abstracts away a lot of the OS specifics, but still lets you write simple code that can either just run pure bash, or do simple/advanced sysadmin things like user management, configure services etc.

    You run them from your local computer, and they SSH in to your hosts. No dependencies expect python.

    Here's an example that does what you need. You need to configure your inventory file, then run the below playbook with ansible-playbook -i inventory ./upgrade_all.yml

    https://www.jeffgeerling.com/blog/2022/ansible-playbook-upgrade-ubuntudebian-servers-and-reboot-if-needed

    Thanked by (3)wankel GeekWanderer skhron
  • wankelwankel OG

    Thanks a lot guys!

    @somik said: You can also make use of bash arrays to pass a list of hosts:

    I'm glad to see that my script-based approach is not totally outdated. Thanks for the array tip :-)

    @terrorgen said: I use NixOS plus colmena.

    I heard good things about NixOS over the years, so it definitely is on the list to try out once.

    @IAmNix said: Ansible (...) is a sysadmin framework

    Ansible may just be the answer I have been looking for, thanks for the introduction!

    Thanked by (1)IAmNix
Sign In or Register to comment.