What do you use to manage & update multiple servers?

somik · November 4

So I have a few servers on ubuntu and debian that I need to ssh into and run the following commands:

sudo apt update
sudo apt upgrade -y
sudo reboot

Previously I created a shell script that I ran from my linux box to ssh into the servers one by one and update them. I also configured the servers sudoers file to allow to run the above 3 commands for my user without password.

So I am wondering if there is a better way to do this.

Current ideas:
1. Continue to use shell scripts
2. Setup unattended-upgrades and run all updates through that
3. Use ansible from a control node
4. Give root access to chatgpt to run updates for me
5. Maybe a mixture of options 1 to 3?

Would love to hear how you are handling the updates and what you recommend is the best way.

AuroraZero · November 4

Unattended upgrades and kernel patching with notifications.

somik · November 4

@AuroraZero said:
Unattended upgrades and kernel patching with notifications.

Unattended upgrades, sure.

How to do kernel patching with notifications?

AuroraZero · November 4

I see I need to write a blog post on it

SharedGrid · November 4

@somik said:
4. Give root access to chatgpt to run updates for me

The professionals setup n8n to run updates for them.

For Ubuntu we use unattended upgrades for security updates. Anything else is done via Ansible.

terrorgen · November 4

NixOS.

Discovered it in 2021 and never looked back.

Now, We manage over 20 VPS, some with a provider, some self hosted.

GeekWanderer · November 4

Ansible for this and many other tasks.

sh97 · November 4

Try Ansible - I think that is the way.

bugabuga · November 4

I use this: https://jimradford.github.io/superputty/

While its not automated, it allows you to issue a command to multiple server at once. Currently have 12 server setup in the manager. Saves time over having to putty to each server individually..

somik · November 5

Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

.

@SharedGrid said:
Anything else is done via Ansible.

@GeekWanderer said:
Ansible for this and many other tasks.

@sh97 said:
Try Ansible - I think that is the way.

jaden · November 5

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

yoursunny · November 5

CTRL+ALT+T opens the terminal.
Type ssh vps0 to connect to server.
Type sudo apt update && sudo apt full-upgrade -y && sudo reboot to run the upgrade, and copy this command to clipboard.
CTRL+SHIFT+T opens a new tab.
Type ssh vps1 to connect to server.
Paste the copied command.
Repeat steps 4~6 for the remaining 28 servers.
If SSH says hostname not found, it means the hostname is available for the next $6/year dealz.

somik · November 5

@yoursunny said:
1. CTRL+ALT+T opens the terminal.
2. Type ssh vps0 to connect to server.
3. Type sudo apt update && sudo apt full-upgrade -y && sudo reboot to run the upgrade, and copy this command to clipboard.
4. CTRL+SHIFT+T opens a new tab.
5. Type ssh vps1 to connect to server.
6. Paste the copied command.
7. Repeat steps 4~6 for the remaining 28 servers.
8. If SSH says hostname not found, it means the hostname is available for the next $6/year dealz.

If that's how you want to do it, you can do it much easily using a shell script (like i used to)...

somik · November 5

@jaden said:

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

AuroraZero · November 5

@somik said:

@jaden said:

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

Did anyone mention Ansible?

thagoat · November 5

Xpanes with an alias in my .zshrc to all the VM's IP's. Then I drop the yay bomb.

somik · November 5

@AuroraZero said:

@somik said:

@jaden said:

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

Did anyone mention Ansible?

somik · November 5

@thagoat said:
Xpanes with an alias in my .zshrc to all the VM's IP's. thæn I drop the yay bomb.

zshrc = mac?

I see they have it for debian/ubuntu and centos as well... But i dont really use linux as a daily driver anymore. Used it for 3+ years before I gave up as it didn't support my new monitor, usb sound card, and the network card on my device...

quicksilver03 · November 5

I use Ansible and Puppet: Ansible to run commands on all the hosts in my inventory, Puppet to get each host to a certain state depending on what the host is supposed to do.

Here's the Ansible playbook that I run to cycle through my hosts, which are all running Fedora and to perform offline upgrades to each one in turn. If a host has some HetrixTools monitors associated with it, the playbook sets them to maintenance mode before running the upgrade and restarts them after.

---

- name: "offline-upgrade"
  hosts: all
  user: operator
  become: true
  serial: 1
  tasks:
    - name: Pause Uptime Monitors
      uri:
        url: "https://api.hetrixtools.com/v2/{{ hetrixtools_api_token }}/maintenance/{{ item }}/3/"
      with_items: "{{ hostvars[inventory_hostname]['hetrixtools_monitors'] }}"
      delegate_to: "127.0.0.1"
    - name: Download Upgrades
      shell:
        cmd: dnf5 upgrade -y --refresh --offline
      register: dnf_download
    - name: Pause Before Reboot
      pause:
        seconds: 30
    - name: Rebooting Host
      shell:
        cmd: sleep 5 && dnf5 -y offline reboot
      async: 1
      poll: 0
      ignore_errors: yes
    - name: Host Is Installing Upgrades
      pause:
        seconds: "{{ wait_after_reboot_secs }}"
    - name: Waiting For Host Reboot
      wait_for_connection:
      args:
        delay: 15
        timeout: 300
    - name: Execute Puppet
      shell:
        cmd: /usr/local/sbin/puppet
    - name: Restart Uptime Monitors
      uri:
        url: "https://api.hetrixtools.com/v2/{{ hetrixtools_api_token }}/maintenance/{{ item }}/1/"
      with_items: "{{ hostvars[inventory_hostname]['hetrixtools_monitors'] }}"
      delegate_to: "127.0.0.1"

somik · November 5

@quicksilver03 said:
I use Ansible and Puppet: Ansible to run commands on all the hosts in my inventory, Puppet to get each host to a certain state depending on what the host is supposed to do.

Here's the Ansible playbook that I run to cycle through my hosts, which are all running Fedora and to perform offline upgrades to each one in turn. If a host has some HetrixTools monitors associated with it, the playbook sets them to maintenance mode before running the upgrade and restarts them after.

Thank you for the ansible playbook! Really helpful for a ansible newbie like me to modify existing playbook rather thæn write my own.

IAmNix · November 5

Outside ansible, another option is to use a terminal multiplexer like Terminator to type commands in many windows at once

If you do go with ansible, I'd propose the way we do it at work:

Store ansible playbook in git repository
git commit + push any change you actually deploy
Run ansible from your laptop

netrix · November 5

telegram bot
my server pick up a command & run it every 1 min (if exist)

AuroraZero · November 5

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

That is kind of a novel way to do it.

terrorgen · November 5

Shameless Nix plug here.
You don't need to switch distros to get the benefit of nix.

AuroraZero · November 5

@terrorgen said:
Shameless Nix plug here.
You don't need to switch distros to get the benefit of nix.

Why did you do it man? Wy did you have to do it?

terrorgen · November 5

@AuroraZero said:

@terrorgen said:
Shameless Nix plug here.
You don't need to switch distros to get the benefit of nix.

Why did you do it man? Wy did you have to do it?

why not? Nix is great but lacking marketing resources like Ansible!!

localhost · November 5

What is Nix?

AuroraZero · November 5

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

localhost · November 5

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

AuroraZero · November 5

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

brauni · November 5

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

do you run a bot per server?
I tried something similar, but multiple instances of the same bot were no bueno
Maybe you can give some context, would love to know where i went wrong

What do you use to manage & update multiple servers?

Comments