What do you use to manage & update multiple servers?

13»

Comments

  • bingobangobongobingobangobongo Hosting Provider

    @IAmNix said:

    @somik said:

    Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

    Anyone know what I am doing wrong?

    This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

    Yep definitely something between versions here… I actually don’t install anything on nodes and have a Debian prep script that runs everything needed, so any blank/base install just werks.

    Will sanitize and post here shortly!

    Rock Solid Web Hosting, VPS & VDS with a Refreshing Approach - Xeon Scalable, DDoS protection and Enterprise Hardware! HostBilby Inc.

  • localhostlocalhost OG

    Currently have my laptop and phone (juice ssh) with keys to most vms

    Realized it hard way a few days ago... if I wanted to transfer files from one vm to another it is next to impossible as all are key based auth and none have keys to login to other

    Curious what is the way to go about...

    Have a decent bunch of idlers so having n idlers keys on m machines seems too much?

  • bingobangobongobingobangobongo Hosting Provider

    @localhost said:
    Currently have my laptop and phone (juice ssh) with keys to most vms

    Realized it hard way a few days ago... if I wanted to transfer files from one vm to another it is next to impossible as all are key based auth and none have keys to login to other

    Curious what is the way to go about...

    Have a decent bunch of idlers so having n idlers keys on m machines seems too much?

    Maybe have a secondary layer of auth/keys for stuff between nodes? Then as part of setup, deploy the key used for inter node work?

    Or Ansible, since noone has mentioned it yet…

    Thanked by (1)localhost

    Rock Solid Web Hosting, VPS & VDS with a Refreshing Approach - Xeon Scalable, DDoS protection and Enterprise Hardware! HostBilby Inc.

  • localhostlocalhost OG

    @bingobangobongo said:

    @localhost said:
    Currently have my laptop and phone (juice ssh) with keys to most vms

    Realized it hard way a few days ago... if I wanted to transfer files from one vm to another it is next to impossible as all are key based auth and none have keys to login to other

    Curious what is the way to go about...

    Have a decent bunch of idlers so having n idlers keys on m machines seems too much?

    Maybe have a secondary layer of auth/keys for stuff between nodes? Then as part of setup, deploy the key used for inter node work?

    Or Ansible, since noone has mentioned it yet…

    Yeah.
    Secondary keys is needed. But wanna keep overhead of number of keys limited.
    How to securely set same key on all nodes?

    I started tinkering with ansible and got my laptop and phone keys added to authorized keys. So there's that. But actual private key sharing over the wire seems trippy?

    Thanked by (1)bingobangobongo
  • WSSWSS OG

    @localhost said:
    Curious what is the way to go about...

    python3 -m HTTPServer

    Thanked by (1)localhost

    My pronouns are like/subscribe.

  • localhostlocalhost OG

    @WSS said:

    @localhost said:
    Curious what is the way to go about...

    python3 -m HTTPServer

    Thats one way I was thinking too.
    Curious if having one key across all makes sense or to do the painful way for x keys on y servers

  • AuroraZeroAuroraZero ModeratorHosting ProviderRetired

    @localhost said: Curious if having one key across all makes sense or to do the painful way for x keys on y servers

    2 keys total, one main one and a backup. If you are the only admin there is no sense making it difficult to manage or prone to human error.

    Thanked by (3)localhost IAmNix bingobangobongo
  • somiksomik OG

    @localhost said:

    @WSS said:

    @localhost said:
    Curious what is the way to go about...

    python3 -m HTTPServer

    Thats one way I was thinking too.
    Curious if having one key across all makes sense or to do the painful way for x keys on y servers

    No, it's never a good thing to share ssh keys.

    I usually go with 3 ways.

    If I want to automate file transfers, I usually create a low level user on the remote server and setup a ssh key between the 2 servers (like from my main server to my backup server).

    If I only need to transfer a lot of files, I use SCP to download it to my desktop and reupload it back.

    If it's just 1 or 2 files, I use: https://github.com/somik123/python3_http_upload_progress

    @IAmNix said:

    @somik said:

    Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

    Anyone know what I am doing wrong?

    This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

    I installed it from ubuntu's repo, when that didn't work, I reinstalled it from python3 pip, but that didn't help either. I guess it's time to either try a proper PPA or install centos 8 and try from there. Thanks for the recommendations!

    @bingobangobongo said:

    @IAmNix said:

    @somik said:

    Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

    Anyone know what I am doing wrong?

    This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

    Yep definitely something between versions here… I actually don’t install anything on nodes and have a Debian prep script that runs everything needed, so any blank/base install just werks.

    Will sanitize and post here shortly!

    Thanks! Looking forward to it!

    Thanked by (1)localhost

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • localhostlocalhost OG

    @somik said:

    @localhost said:

    @WSS said:

    @localhost said:
    Curious what is the way to go about...

    python3 -m HTTPServer

    Thats one way I was thinking too.
    Curious if having one key across all makes sense or to do the painful way for x keys on y servers

    No, it's never a good thing to share ssh keys.

    I usually go with 3 ways.

    If I want to automate file transfers, I usually create a low level user on the remote server and setup a ssh key between the 2 servers (like from my main server to my backup server).

    If I only need to transfer a lot of files, I use SCP to download it to my desktop and reupload it back.

    If it's just 1 or 2 files, I use: https://github.com/somik123/python3_http_upload_progress

    Interesting
    Well file transfer is definitely not automated.

    And its not just transfer. I may need to quickly ssh into another box right from there... I know I can connect organically from my laptop. But since inam already on server 1 so if I need to check something in server 2 it should be easy, right?

    Thanks for the quick http file server. Will bookmark it

  • nullnotherenullnothere OG

    I've not been following this fully, but a few quick hints:

    1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
    2. You have an ssh key for logins to connect from A->B and A->C
    3. You want to transfer some file(s) from B->C
    4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
    5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
    6. man ssh-agent and ssh-add for options/security etc. RTFM.

    Profit?

    Thanked by (1)localhost
  • localhostlocalhost OG

    @nullnothere said:
    I've not been following this fully, but a few quick hints:

    1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
    2. You have an ssh key for logins to connect from A->B and A->C
    3. You want to transfer some file(s) from B->C
    4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
    5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
    6. man ssh-agent and ssh-add for options/security etc. RTFM.

    Profit?

    I think ssh-agent forwarding may work. Thanks

  • somiksomik OG

    @localhost said:

    @nullnothere said:
    I've not been following this fully, but a few quick hints:

    1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
    2. You have an ssh key for logins to connect from A->B and A->C
    3. You want to transfer some file(s) from B->C
    4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
    5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
    6. man ssh-agent and ssh-add for options/security etc. RTFM.

    Profit?

    I think ssh-agent forwarding may work. Thanks

    ssh bastion host?

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • localhostlocalhost OG

    @somik said:

    @localhost said:

    @nullnothere said:
    I've not been following this fully, but a few quick hints:

    1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
    2. You have an ssh key for logins to connect from A->B and A->C
    3. You want to transfer some file(s) from B->C
    4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
    5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
    6. man ssh-agent and ssh-add for options/security etc. RTFM.

    Profit?

    I think ssh-agent forwarding may work. Thanks

    ssh bastion host?

    That too...
    Any recommendations?

  • somiksomik OG

    @localhost said:

    @somik said:
    ssh bastion host?

    That too...
    Any recommendations?

    DIY solution without any software bloats or extra security risks?
    https://smallstep.com/blog/diy-ssh-bastion-host/

    Or you can go with one of the more popular software solutions:
    https://goteleport.com/blog/ssh-bastion-host/

    Or go with a proper zero trust network with bastionXP:
    https://www.bastionxp.com/docs/guide/

    Thanked by (2)localhost IAmNix

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • IAmNixIAmNix

    @nullnothere said:
    I've not been following this fully, but a few quick hints:

    1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
    2. You have an ssh key for logins to connect from A->B and A->C
    3. You want to transfer some file(s) from B->C
    4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
    5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
    6. man ssh-agent and ssh-add for options/security etc. RTFM.

    The ssh agent forwarding option is -A.

    So the commands would look like this:

    user@laptop:~$ ssh -A server-1
user@server-1:~$ scp my-files/ server-2:

    Only do this if you trust server-1. It essentially shares access to all ssh keys on your laptop. As the man ssh says:

    Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's Unix-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent. A safer alternative may be to use a jump host (see -J).

  • bingobangobongobingobangobongo Hosting Provider

    Finally posting my Ansible template for Debian setup... Sorry for the delay!


    Codie mc Coderson 
    ---
- name: Initial Debian Server Setup
  hosts: all
  gather_facts: no

  # Prompt for credentials at the very start.
  vars:
    # Automatically read the public key from the control node's /.ssh/id_ecdsa.pub
    authorized_pub_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ecdsa.pub') }}"

  vars_prompt:
    - name: "username"
      prompt: "Enter the username to create"
      private: no
    - name: "password"
      prompt: "Enter the password for the new user"
      private: yes
      confirm: yes

  pre_tasks:
    - name: Install Python (required for all other Ansible modules)
      raw: |
        if ! command -v python3 &> /dev/null; then
          apt update && apt install -y python3
        fi
      become: yes
      changed_when: false

    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Upgrade all installed packages (dist-upgrade)
      apt:
        upgrade: dist

    - name: Check if sudo is installed
      command: which sudo
      register: sudo_check
      ignore_errors: yes

    - name: Install sudo if not present
      apt:
        name: sudo
        state: present
        update_cache: yes
      when: sudo_check.rc != 0

    - name: Install community.general collection
      ansible.builtin.command: ansible-galaxy collection install community.general
      delegate_to: localhost
      run_once: true
      changed_when: false

    - name: Gather facts now that everything is set up
      setup:

  tasks:
    - block:
        - name: Install generally required packages
          apt:
            name:
              - curl
              - zip
              - unzip
              - git
              - smartmontools
              - nvme-cli
              - htop
              - iperf3
              - dnsutils
              - net-tools
              - whois
            state: present
            update_cache: yes

        - name: Hash the password using mkpasswd
          command: "mkpasswd --method=SHA-512 {{ password | quote }}"
          register: hashed_password
          no_log: true

        - name: Create the user with a hashed password
          user:
            name: "{{ username }}"
            password: "{{ hashed_password.stdout }}"
            shell: /bin/bash
            state: present
            groups: sudo
            append: yes
            createhome: yes
          become: yes

        - name: Pause for 3 seconds to ensure user is registered
          pause:
            seconds: 3

        - name: Ensure the user's home directory exists and set ownership
          file:
            path: "/home/{{ username }}"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0755'

        - name: Ensure .ssh directory exists with correct permissions
          file:
            path: "/home/{{ username }}/.ssh"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0700'

        - name: Add authorized SSH key for the new user
          authorized_key:
            user: "{{ username }}"
            key: "{{ authorized_pub_key }}"
            state: present
            path: "/home/{{ username }}/.ssh/authorized_keys"

        - name: Set correct permissions on the authorized_keys file
          file:
            path: "/home/{{ username }}/.ssh/authorized_keys"
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0600'

        - name: Allow the user passwordless sudo in sudoers.d
          copy:
            dest: "/etc/sudoers.d/{{ username }}"
            content: "{{ username }} ALL=(ALL) NOPASSWD: ALL\n"
            mode: '0440'

        - name: Remove all files in sshd_config.d directory
          command: find /etc/ssh/sshd_config.d/ -type f -delete
          changed_when: true

        - name: Configure SSH to disable password authentication for sudo/admin users
          copy:
            dest: /etc/ssh/sshd_config.d/10-disable_password_for_admins.conf
            content: |
              Match Group sudo,admin
                  PasswordAuthentication no
            mode: '0644'
          notify: restart ssh

        - name: Ensure PermitRootLogin is set to prohibit-password in sshd_config
          lineinfile:
            path: /etc/ssh/sshd_config
            regexp: '^#?PermitRootLogin\s+.*$'
            line: 'PermitRootLogin prohibit-password'
            state: present
            backup: yes
          notify: restart ssh

        - name: Install fail2ban package
          apt:
            name: fail2ban
            state: present
            update_cache: yes

        - name: Configure fail2ban with custom settings and SSH protection
          copy:
            dest: /etc/fail2ban/jail.d/custom.conf
            content: |
              [DEFAULT]
              ignoreip = 127.0.0.1
              bantime = 1h
              findtime = 10m
              maxretry = 3
              banaction = ufw

              [sshd]
              enabled = true
              port    = ssh
              filter  = sshd
              logpath = /var/log/auth.log
            mode: '0644'
          notify: restart fail2ban

        - name: Install ufw package
          apt:
            name: ufw
            state: present
            update_cache: yes

        - name: Allow SSH through UFW
          community.general.ufw:
            rule: allow
            name: OpenSSH
          notify: restart ufw

        - name: Deny RPC through UFW
          community.general.ufw:
            rule: deny   
            port: '111'
          notify: restart ufw

        - name: Enable UFW on startup
          community.general.ufw:
            state: enabled
          notify: restart ufw

      rescue:
        - debug:
            msg: "An error occurred during provisioning. Please check the logs above."
        - fail:
            msg: "Provisioning failed."

    - debug:
        msg: "Provisioning complete. All tasks executed successfully."

    - name: Flush all pending handlers
      meta: flush_handlers

    - name: Prompt for reboot confirmation
      pause:
        prompt: "Server setup is complete. Do you want to reboot the server now? (y/n)"
      register: reboot_response

    - name: Reboot the server
      reboot:
        reboot_timeout: 300
      when: reboot_response.user_input | lower in ['y', 'yes']

    - name: Display message if reboot was skipped
      debug:
        msg: "Reboot skipped. You may want to reboot manually later to ensure all changes take effect."
      when: reboot_response.user_input | lower not in ['y', 'yes']

  handlers:
    - name: restart ssh
      service:
        name: ssh
        state: restarted

    - name: restart fail2ban
      service:
        name: fail2ban
        state: restarted

    - name: restart ufw
      service:
        name: ufw
        state: restarted
    Thanked by (1)cochon

    Rock Solid Web Hosting, VPS & VDS with a Refreshing Approach - Xeon Scalable, DDoS protection and Enterprise Hardware! HostBilby Inc.

  • somiksomik OG

    @bingobangobongo said:
    Finally posting my Ansible template for Debian setup... Sorry for the delay!


    Codie mc Coderson 
    ---
- name: Initial Debian Server Setup
  hosts: all
  gather_facts: no

  # Prompt for credentials at the very start.
  vars:
    # Automatically read the public key from the control node's /.ssh/id_ecdsa.pub
    authorized_pub_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ecdsa.pub') }}"

  vars_prompt:
    - name: "username"
      prompt: "Enter the username to create"
      private: no
    - name: "password"
      prompt: "Enter the password for the new user"
      private: yes
      confirm: yes

  pre_tasks:
    - name: Install Python (required for all other Ansible modules)
      raw: |
        if ! command -v python3 &> /dev/null; thæn
          apt update && apt install -y python3
        fi
      become: yes
      changed_when: false

    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Upgrade all installed packages (dist-upgrade)
      apt:
        upgrade: dist

    - name: Check if sudo is installed
      command: which sudo
      register: sudo_check
      ignore_errors: yes

    - name: Install sudo if not present
      apt:
        name: sudo
        state: present
        update_cache: yes
      when: sudo_check.rc != 0

    - name: Install community.general collection
      ansible.builtin.command: ansible-galaxy collection install community.general
      delegate_to: localhost
      run_once: true
      changed_when: false

    - name: Gather facts now that everything is set up
      setup:

  tasks:
    - block:
        - name: Install generally required packages
          apt:
            name:
              - curl
              - zip
              - unzip
              - git
              - smartmontools
            - nvme-cli
              - htop
            - iperf3
            - dnsutils
            - net-tools
              - whois
            state: present
            update_cache: yes

        - name: Hash the password using mkpasswd
          command: "mkpasswd --method=SHA-512 {{ password | quote }}"
          register: hashed_password
          no_log: true

        - name: Create the user with a hashed password
          user:
            name: "{{ username }}"
            password: "{{ hashed_password.stdout }}"
            shell: /bin/bash
            state: present
            groups: sudo
            append: yes
            createhome: yes
          become: yes

        - name: Pause for 3 seconds to ensure user is registered
          pause:
            seconds: 3

        - name: Ensure the user's home directory exists and set ownership
          file:
            path: "/home/{{ username }}"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0755'

        - name: Ensure .ssh directory exists with correct permissions
          file:
            path: "/home/{{ username }}/.ssh"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0700'

        - name: Add authorized SSH key for the new user
          authorized_key:
            user: "{{ username }}"
            key: "{{ authorized_pub_key }}"
            state: present
            path: "/home/{{ username }}/.ssh/authorized_keys"

        - name: Set correct permissions on the authorized_keys file
          file:
            path: "/home/{{ username }}/.ssh/authorized_keys"
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0600'

        - name: Allow the user passwordless sudo in sudoers.d
          copy:
            dest: "/etc/sudoers.d/{{ username }}"
            content: "{{ username }} ALL=(ALL) NOPASSWD: ALL\n"
            mode: '0440'

        - name: Remove all files in sshd_config.d directory
          command: find /etc/ssh/sshd_config.d/ -type f -delete
          changed_when: true

        - name: Configure SSH to disable password authentication for sudo/admin users
          copy:
            dest: /etc/ssh/sshd_config.d/10-disable_password_for_admins.conf
            content: |
              Match Group sudo,admin
                  PasswordAuthentication no
            mode: '0644'
          notify: restart ssh

        - name: Ensure PermitRootLogin is set to prohibit-password in sshd_config
          lineinfile:
            path: /etc/ssh/sshd_config
            regexp: '^#?PermitRootLogin\s+.*$'
            line: 'PermitRootLogin prohibit-password'
            state: present
            backup: yes
          notify: restart ssh

        - name: Install fail2ban package
          apt:
            name: fail2ban
            state: present
            update_cache: yes

        - name: Configure fail2ban with custom settings and SSH protection
          copy:
            dest: /etc/fail2ban/jail.d/custom.conf
            content: |
              [DEFAULT]
              ignoreip = 127.0.0.1
              bantime = 1h
              findtime = 10m
              maxretry = 3
              banaction = ufw

              [sshd]
              enabled = true
              port    = ssh
              filter  = sshd
              logpath = /var/log/auth.log
            mode: '0644'
          notify: restart fail2ban

        - name: Install ufw package
          apt:
            name: ufw
            state: present
            update_cache: yes

        - name: Allow SSH through UFW
          community.general.ufw:
            rule: allow
            name: OpenSSH
          notify: restart ufw

        - name: Deny RPC through UFW
          community.general.ufw:
            rule: deny   
            port: '111'
          notify: restart ufw

        - name: Enable UFW on startup
          community.general.ufw:
            state: enabled
          notify: restart ufw

      rescue:
        - debug:
            msg: "An error occurred during provisioning. Please check the logs above."
        - fail:
            msg: "Provisioning failed."

    - debug:
        msg: "Provisioning complete. All tasks executed successfully."

    - name: Flush all pending handlers
      meta: flush_handlers

    - name: Prompt for reboot confirmation
      pause:
        prompt: "Server setup is complete. Do you want to reboot the server now? (y/n)"
      register: reboot_response

    - name: Reboot the server
      reboot:
        reboot_timeout: 300
      when: reboot_response.user_input | lower in ['y', 'yes']

    - name: Display message if reboot was skipped
      debug:
        msg: "Reboot skipped. You may want to reboot manually later to ensure all changes take effect."
      when: reboot_response.user_input | lower not in ['y', 'yes']

  handlers:
    - name: restart ssh
      service:
        name: ssh
        state: restarted

    - name: restart fail2ban
      service:
        name: fail2ban
        state: restarted

    - name: restart ufw
      service:
        name: ufw
        state: restarted

    Wow... I'll need this weekend to process this...

    Thanks for the detailed ansible template!

    Thanked by (1)bingobangobongo

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • SpeedBusSpeedBus Hosting ProviderOG

    Found this on reddit a while ago,
    https://www.reddit.com/r/selfhosted/comments/1nlth9k/finally_built_patchmon_my_linux_updates

    GitHub: https://github.com/PatchMon/PatchMon

    Thanked by (1)bingobangobongo

    CrownCloud - Internet Services | Los Angeles, California | Frankfurt, Germany | Amsterdam, The Netherlands | Atlanta, Georgia | Miami, Florida

Sign In or Register to comment.

© LowEndSpirit 2025

ipv6 ready