What do you use to manage & update multiple servers?

localhost · November 5

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

So what kind of things will calm you down?

AuroraZero · November 5

@localhost said:

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

So what kind of things will calm you down?

I am calm man I have not banned anyone since my second term that level of calm is extraordinary.

terrorgen · November 5

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

Why is nix weird?

AuroraZero · November 5

@terrorgen said:

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

Why is nix weird?

Real Q is why isn't it?

terrorgen · November 5

@AuroraZero said:

@terrorgen said:

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

Why is nix weird?

Real Q is why isn't it?

Speaking about nix as a package-manager-as-code, it is like ansible with its own repo, and more. Nix is a programming language tailored for reproducibility of system environments.

Nix was already out there before Ansible was even a thing. But Ansible has marketing budget, so...

bingobangobongo · November 6

You should definitely check out Ansible! It's super powerful and very amazing - much gud.

somik · November 6

@IAmNix said:
Outside ansible, another option is to use a terminal multiplexer like Terminator to type commands in many windows at once

If you do go with ansible, I'd propose the way we do it at work:

Store ansible playbook in git repository

git commit + push any change you actually deploy

Run ansible from your laptop

Wont that expose my server's IP and ssh port? Or do you mean self hosted git repository or private github repo?

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

Ya, tried that, but gave up quickly. I would rather run it via cron thæn use telegram bot message polling.

@brauni said:

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

do you run a bot per server?
I tried something similar, but multiple instances of the same bot were no bueno
Maybe you can give some context, would love to know where i went wrong

No, you have 1 centralized bot and poll the bot for messages (not webhook). So basically you request this url every 1 min:
https://api.telegram.org//getUpdates

Note that you NEED to have webhooks disabled for the bot for this to work, so go ahead and create a new bot using telegram botfather for this. Once you send any message to the telegram bot, it'll show up if you open the api getUpdates as json. You can parse it to use the same bot to send commands to all servers.

@bingobangobongo said:
You should definitely check out Ansible! It's super powerful and very amazing - much gud.

Weird... I dont think anyone mentioned Ansible yet. Is it something new?

somik · November 6

And yes, I am completely skipping over everything related to nix

AuroraZero · November 6

@somik said:
And yes, I am completely skipping over everything related to nix

Chicken!!!!

terrorgen · November 6

@somik said:
And yes, I am completely skipping over everything related to nix

bo-hood

no balls

somik · November 6

@AuroraZero said:

@somik said:
And yes, I am completely skipping over everything related to nix

Chicken!!!!

@terrorgen said:

@somik said:
And yes, I am completely skipping over everything related to nix

bo-hood

no balls

terrorgen · November 6

It's really not that bad...

somik · November 6

@terrorgen said:
It's really not that bad...

You sure? People complained about snaps when ubuntu introduced it. Now you want to discuss NixOS!!!

I mean jJust look at the screenshots...

Regular linux (Linux Mint) vs NixOS:

terrorgen · November 6

You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

somik · November 6

@terrorgen said:
You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

Ofcourse! Haven't you heard about judging a book by it's cover? It's where the title and author are printed.

Nix, I'm guessing, it's what nixos uses? Like a snap or flatpac lookalike?

cochon · November 6

Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

AuroraZero · November 6

You all know I am just picking on nix right? It is a good tool that does it job. No need for the hate I am just trying to have fun and make people laugh.

On that note did anyone mention Ansible?

WSS · November 6

I used ansible when working for providers. My own set are so diverse that it makes no sense to automate anything except for my cluster of nameservers, because everything else is completely purpose-built for that specific task.

terrorgen · November 6

@somik said:

@terrorgen said:
You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

Ofcourse! Haven't you heard about judging a book by it's cover? It's where the title and author are printed.

Nix, I'm guessing, it's what nixos uses? Like a snap or flatpac lookalike?

it is like snap and ansible had a baby that was born before both existed.

somik · November 7

@cochon said:
Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

You mean like deleting the only VM that stored all your ssh keys that you used to access all of your other servers and not having any backups?

No, that never happened to me. I am not that stupid.

Oh, you are asking why I had to recreate and reinstall OS on all of my servers in 2024? That's because I felt it was time for a refresh; and since I had backups of my data in my external drive, why not!

Definitely not because I am stupid and deleted the only VM that ran on my windows laptop that had the only pair of ssh keys that I used to access all the other servers...

@AuroraZero said:
On that note did anyone mention Ansible?

No, I dont believe anyone had mentioned that...

@WSS said:
I used ansible when working for providers. My own set are so diverse that it makes no sense to automate anything except for my cluster of nameservers, because everything else is completely purpose-built for that specific task.

Sounds like I have what I need to work on this weekend... Ansible...

Now where should I set it up? A raspberry pi?

@terrorgen said:

@somik said:

@terrorgen said:
You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

Ofcourse! Haven't you heard about judging a book by it's cover? It's where the title and author are printed.

Nix, I'm guessing, it's what nixos uses? Like a snap or flatpac lookalike?

it is like snap and ansible had a baby that was born before both existed.

Sounds like fun! I'll look into it when I have more time. If it works on mac, i might be able to use it to install and switch between all the Java versions I need for work...

terrorgen · November 7

Nix works on Macs too! Check out nix-darwin.

AuroraZero · November 7

@terrorgen said:
Nix works on Macs too! Check out nix-darwin.

Son of a they got another one

somik · November 7

@AuroraZero said:

@terrorgen said:
Nix works on Macs too! Check out nix-darwin.

Son of a they got another one

Well, on my office mac, we are already using "uv" to run python as we need to switch between different envs. So why not give nix a try. If i like it (and the corporate IT overlords allows it) all good. Otherwise, just try something else.

bingobangobongo · November 7

@somik said: Now where should I set it up? A raspberry pi?

Definitely in the same space that the old VM you deleted took up on your laptop. It's full circle - she's home again!

cochon · November 7

@somik said:

@cochon said:
Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

You mean like deleting the only VM that stored all your ssh keys that you used to access all of your other servers and not having any backups?

No I don’t mean that at all, I was thinking more of making an accidental config change (maybe a typo) that locks you out of remote admin completely. I then still have a pull update mechanism to fall back on.

No, that never happened to me. I am not that stupid.

To be honest that’s not a great attitude for a sysadmin. Where I come from we have a saying: ‘pride comes before a fall‘. I also think I’m not that stupid, but I always factor in that one day I might be

somik · November 7

@cochon said:

@somik said:

@cochon said:
Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

You mean like deleting the only VM that stored all your ssh keys that you used to access all of your other servers and not having any backups?

No I don’t mean that at all, I was thinking more of making an accidental config change (maybe a typo) that locks you out of remote admin completely. I thæn still have a pull update mechanism to fall back on.

No, that never happened to me. I am not that stupid.

To be honest that’s not a great attitude for a sysadmin. Where I come from we have a saying: ‘pride comes before a fall‘. I also think I’m not that stupid, but I always factor in that one day I might be

Ah, I see that went over your head...

Some background info for you.

In 2024, i lost access to all my servers cause the only ssh key was in a VM on my laptop that I deleted when I reformatted my laptop. My post was meant to be a sarcastic reminder to myself for that mistake...

hornet · November 7

@GeekWanderer said: Ansible

This is the way.

AuroraZero · November 7

I believe it would be a dereliction of my duty if I did not mention Ansible.

somik · November 8

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?


fatal: [dev]: FAILED! => 
{
  "ansible_facts": {},
  "changed": false,
  "failed_modules": {
    "ansible.legacy.setup": {
      "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
      },
      "exception": "Traceback (most recent call last):\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 102, in <module>\r\n _ansiballz_main()\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 94, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 37, in invoke_module\r\n from ansible.module_utils import basic\r\n File \"/tmp/ansible_ansible.legacy.setup_payload_vbni3qiv/ansible_ansible.legacy.setup_payload.zip/ansible/module_utils/basic.py\", line 176, in <module>\r\nModuleNotFoundError: No module named 'ansible.module_utils.six.moves'\r\n",
      "failed": true,
      "module_stderr": "Shared connection to 10.10.0.50 closed.\r\n",
      "module_stdout": "Traceback (most recent call last):\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 102, in <module>\r\n _ansiballz_main()\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 94, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 37, in invoke_module\r\n from ansible.module_utils import basic\r\n File \"/tmp/ansible_ansible.legacy.setup_payload_vbni3qiv/ansible_ansible.legacy.setup_payload.zip/ansible/module_utils/basic.py\", line 176, in <module>\r\nModuleNotFoundError: No module named 'ansible.module_utils.six.moves'\r\n",
      "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
      "rc": 1
    }
  },
  "msg": "The following modules failed to execute: ansible.legacy.setup\n"
}

My ansible file:

- name: Update VMs
  hosts: vms
  gather_facts: yes
  become: yes
  tasks:
    - name: Ping host
      ping:
      register: ping_result
      ignore_errors: yes

    - name: Update APT cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Upgrade packages
      apt:
        upgrade: dist
        autoremove: yes
        autoclean: yes

    - name: Reboot if required
      reboot:
        msg: "Rebooting VM after upgrade"
        pre_reboot_delay: 5
        post_reboot_delay: 10
        test_command: uptime

- name: Wait for VMs to be back online
  hosts: localhost
  gather_facts: no
  tasks:
    - name: Pause 2 minutes for VM reboot
      pause:
        minutes: 2

    - name: Wait for each VM SSH
      wait_for:
        host: "{{ item }}"
        port: "{{ hostvars[item].ansible_port | default(22) }}"
        timeout: 300
      loop: "{{ groups['vms'] }}"

IAmNix · November 9

@somik said:

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?

This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

@somik said:

Wont that expose my server's IP and ssh port? Or do you mean self hosted git repository or private github repo?

Not unless you want to share the git repo with other people - I was assuming this was 100% for personal use, and that you already have SSH access to some server to store the git repo somehow.
To be clear this git shenanigans is only useful for you to sync your ansible code between different personal machines, and as a simple backup. All it does is copy the code to a server somewhere.

For shared usage I would just use a public git host with a private repository, like codeberg.org.

(This SSH trick works because a git repository is technically just any folder with a .git subfolder in it. So we can use any protocol to sync the changes.)

How to setup an ansible git repo over SSH:

On your remote machine:

mkdir my-ansible; cd my-ansible
git init
# git commit something just to test with

On your laptop:

git clone ssh://user@remote-machine/path/to/my-ansible
cd my-ansible
git log # You should see your test commit
# git commit something
git push # <-- this will automatically push over SSH and changes will show up in the remote folder
git pull # <-- this will automatically fetch any changes from the remote folder

What do you use to manage & update multiple servers?

Comments