VPS security in the wake of the Virtualizor ransomware attacks

edited January 31 in Requests

I have been reading about the ransomware attacks that hit Cloudcone and now maybe Hostslick. Seems like there is some exploit in Virtualizor now as it's hitting different providers and not just Cloudcone's custom backend like someone said here.

It looks like all these hackers wanted was to ransom the data, but this presents a security issue, like what if they were able to access and leak the personal data (edit, lol sorry forgot to finish this thought!)

There was previously a thread on setting up LUKS encryption on my VPS which I have done. However I am now feeling quite dismayed seeing that anyone with hypervisor access can trivially decrypt the key from the running ram of the machine, I would assume also the ransomware hackers would have access to this if they had hypervisor access.

The only mitigation against being able to pull the RAM key if I understand correctly is if the VPS provider has a CPU that supports encrypted RAM and also enables this CPU flag in the VPS themselves. I don't think it is very common for VPS providers to support encrypted ram but if anyone has a list let me know.

I was recently playing around with systemd-homed on my home machines which keeps the home dir encrypted, even to root. I haven't tried it on VPS but I wonder if that would provide an additional layer of security against something like this attack, as maybe they would grab the ram key, but my user's home directory would still be encrypted inside that image. Actually I wonder if that's the case and systemd-homed works to encrypt the user directory would there be any benefits in using LUKS for the full drive over systemd-homed on the user's home?

Just feeling a little bummed right now that we don't have more protection for our data being leaked against these types of attacks even if we did everything right. Maybe this could push more providers to move towards encrypted ram, etc.

I don't know a bunch about this area and I'm still learning so please let me know if there's anything else I'm missing about how to further secure private data on a VPS if an adversary got hypervisor access.

Comments

  • MikeAMikeA Hosting ProviderOG

    @lowendmeow said: However I am now feeling quite dismayed seeing that anyone with hypervisor access can trivially decrypt the key from the running ram of the machine, I would assume also the ransomware hackers would have access to this if they had hypervisor access.

    Honestly actors compromising systems are probably not going to go through the hassle to do that on an individual VM basis. They just want money. Or to screw companies that made them mad.

    Thanked by (3)skorous bikegremlin IAmNix
  • @MikeA said:

    @lowendmeow said: However I am now feeling quite dismayed seeing that anyone with hypervisor access can trivially decrypt the key from the running ram of the machine, I would assume also the ransomware hackers would have access to this if they had hypervisor access.

    Honestly actors compromising systems are probably not going to go through the hassle to do that on an individual VM basis. They just want money. Or to screw companies that made them mad.

    Yes, I understand that realistically but it still makes me nervous if I had sensitive data on the VPS and was wondering how much LUKS/systemd-homed could help in this case. Also just because I'm interested in privacy and understanding and being informed about the realistic privacy limits I'd face using a VPS vs a dedicated server and what ways could I theoretically mitigate this. There's also scenarios for example where say a client decided to hold me liable for their data being accessed by a bad actor. If I could say, well I did LUKS encyption, also encrypted home directory, did X Y Z hardening, but at the end of the day if a bad actor accesses the VPS provider and hypervisor there's only so much I can protect against, it seems like I'd be positioned better legally. Probably above my paygrade but I'm wondering what the answer would be

  • @lowendmeow said:
    It looks like all these hackers wanted was to ransom the data, but this presents a security issue, like what if they

    What if they what?!?

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • @somik said:

    @lowendmeow said:
    It looks like all these hackers wanted was to ransom the data, but this presents a security issue, like what if they

    What if they what?!?

    Lol sorry to keep you hanging it's been a long day. Meant to say what if they were able to access and leak the personal data

  • tentortentor Hosting Provider

    If you are that much concerned about your data, I would recommend considering getting a physical server, doing LUKS here and never again thinking about malicious hypervisors.

    If you want to complicate your life and use VM, you might want to look into confidential computing. Good starting point: https://virtee.io/

    Thanked by (1)oloke

    Check our KVM VPS plans in 🇵🇱 Warsaw, Poland and 🇸🇪 Stockholm, Sweden

  • @tentor said:
    If you are that much concerned about your data, I would recommend considering getting a physical server, doing LUKS here and never again thinking about malicious hypervisors.

    If you want to complicate your life and use VM, you might want to look into confidential computing. Good starting point: https://virtee.io/

    Honestly it's more of just a thought experiment for me! Because I want to understand the technology and its limits.

    Whoa thanks so much for this link! So this could be the future of truly secure VPS? Is it in a usable state? Do any providers provide this? What a cool idea! I hope it takes off

  • somiksomik OG
    edited January 31

    @lowendmeow said:

    @tentor said:
    If you are that much concerned about your data, I would recommend considering getting a physical server, doing LUKS here and never again thinking about malicious hypervisors.

    If you want to complicate your life and use VM, you might want to look into confidential computing. Good starting point: https://virtee.io/

    Honestly it's more of just a thought experiment for me! Because I want to understand the technology and its limits.

    Whoa thanks so much for this link! So this could be the future of truly secure VPS? Is it in a usable state? Do any providers provide this? What a cool idea! I hope it takes off

    If it's a VPS, at minimum, there are 3 possible attack targets; the VPS control panel, the provider and your setup.

    If it's a dedicated server, the possible attack target comes down to your provider (if they offer kvm over ip), and your setup. So if it's a server without kvm over ip, the only attack target is your setup. So always better to go with a dedicated server for better security. Use a VPS only for development or testing things.

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • skorousskorous OGSenpai

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

  • @skorous said:

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

    Well, you'll probably be fine with top cloud providers. Go with AWS or OCI or GC or something like that.

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • 208.92.227.181:443
    Possibel this

  • @skorous said:

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

    What is the attack vector on dedicated devices if they have bad IPMI or KVM over IP? Would that be interfering with my unencrypted boot partition to keylog luks password etc?

  • somiksomik OG
    edited February 1

    @lowendmeow said:

    @skorous said:

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

    What is the attack vector on dedicated devices if they have bad IPMI or KVM over IP? Would that be interfering with my unencrypted boot partition to keylog luks password etc?

    If IPMI / BMC / KVM-over-IP is compromised or malicious, you should assume full physical-equivalent access to the machine.

    That includes:

    • Pre-boot control: IPMI sits below the OS. A malicious BMC can tamper with the boot process before your kernel or initramfs ever runs.
    • LUKS password capture: KVM-over-IP literally sees (and can inject) keyboard input. Capturing the LUKS passphrase at boot is trivial.
    • Boot chain modification:

      • Modify the unencrypted /boot partition (kernel, initramfs, cmdline)
      • Inject a malicious initramfs that exfiltrates keys after unlock
      • Persist rootkits that survive reboots
    • Virtual media attacks: IPMI can mount “virtual CD/USB” and boot arbitrary images without you noticing.

    • DMA-style attacks: Some BMCs can read/write system memory directly.
    • Firmware persistence: A compromised BMC can re-infect the host OS even after a clean reinstall.

    Even if /boot is encrypted (or Secure Boot is enabled), a malicious BMC can still keylog or screen-scrape the passphrase during unlock unless you have a trusted input path—which you don’t over KVM-IP.

    If the IPMI is bad, assume the box is owned. LUKS protects against lost disks, not malicious out-of-band management.

    Edit: Post reformatted with LLM to make it easier to understand.

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • Using someone else's computer is okay-ish to store_ sensitive data_ (as in "life threatening"), as long as it's 1/ encrypted (efficiently, beforehand) and 2/ as backup (i.e. never to be decrypted there). Then, nasty guys can wipe/leak/ransomware/screw the data any way they want: you couldn't care less.

    Otherwise it's simply a disaster waiting to happen...

    Thanked by (1)IAmNix

    Happy customer at AlexHost, AxusHost, Bakker-IT, Host-C, Ionos, Veesp + NanoKVM ftw.

  • @Shot² said:
    Using someone else's computer is okay-ish to store_ sensitive data_ (as in "life threatening"), as long as it's 1/ encrypted (efficiently, beforehand) and 2/ as backup (i.e. never to be decrypted there). Then, nasty guys can wipe/leak/ransomware/screw the data any way they want: you couldn't care less.

    Otherwise it's simply a disaster waiting to happen...

    You mean e2e only? Because what I'm learning is that LUKS on a VPS provider that doesn't have non encrypted ram, as long as it's powered on could theoretically have VPS imaged, ram dumped and then LUKS is trivial to unlock

  • @somik said:

    @lowendmeow said:

    @skorous said:

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

    What is the attack vector on dedicated devices if they have bad IPMI or KVM over IP? Would that be interfering with my unencrypted boot partition to keylog luks password etc?

    If IPMI / BMC / KVM-over-IP is compromised or malicious, you should assume full physical-equivalent access to the machine.

    That includes:

    • Pre-boot control: IPMI sits below the OS. A malicious BMC can tamper with the boot process before your kernel or initramfs ever runs.
    • LUKS password capture: KVM-over-IP literally sees (and can inject) keyboard input. Capturing the LUKS passphrase at boot is trivial.
    • Boot chain modification:

      • Modify the unencrypted /boot partition (kernel, initramfs, cmdline)
      • Inject a malicious initramfs that exfiltrates keys after unlock
      • Persist rootkits that survive reboots
    • Virtual media attacks: IPMI can mount “virtual CD/USB” and boot arbitrary images without you noticing.

    • DMA-style attacks: Some BMCs can read/write system memory directly.
    • Firmware persistence: A compromised BMC can re-infect the host OS even after a clean reinstall.

    Even if /boot is encrypted (or Secure Boot is enabled), a malicious BMC can still keylog or screen-scrape the passphrase during unlock unless you have a trusted input path—which you don’t over KVM-IP.

    If the IPMI is bad, assume the box is owned. LUKS protects against lost disks, not malicious out-of-band management.

    Edit: Post reformatted with LLM to make it easier to understand.

    Good reminder about IPMI I don't expect it's bad on any of my providers but I do go in and change the LUKS password after I need to use it to set up or log in.

    I don't know much about this but when doing some research with AI it was suggesting ukify and sbctl. I don't know much about using secure boot on VPS and honestly skip it on my personal computers because I've been installing linux on PCs long enough that I knew it would interfere with boot, I guess now some OS do have signatures that can work with secure boot and not all. I guess the idea sounds like you do encrypt boot and then it would let you know if something was tampered with in this case? No idea if this can be done in a VPS environment and especially since I don't know how the setup with dropbear unlock works if boot is encrypted

  • @somik said:

    @lowendmeow said:

    @skorous said:

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

    What is the attack vector on dedicated devices if they have bad IPMI or KVM over IP? Would that be interfering with my unencrypted boot partition to keylog luks password etc?

    If IPMI / BMC / KVM-over-IP is compromised or malicious, you should assume full physical-equivalent access to the machine.

    That includes:

    • Pre-boot control: IPMI sits below the OS. A malicious BMC can tamper with the boot process before your kernel or initramfs ever runs.
    • LUKS password capture: KVM-over-IP literally sees (and can inject) keyboard input. Capturing the LUKS passphrase at boot is trivial.
    • Boot chain modification:

      • Modify the unencrypted /boot partition (kernel, initramfs, cmdline)
      • Inject a malicious initramfs that exfiltrates keys after unlock
      • Persist rootkits that survive reboots
    • Virtual media attacks: IPMI can mount “virtual CD/USB” and boot arbitrary images without you noticing.

    • DMA-style attacks: Some BMCs can read/write system memory directly.
    • Firmware persistence: A compromised BMC can re-infect the host OS even after a clean reinstall.

    Even if /boot is encrypted (or Secure Boot is enabled), a malicious BMC can still keylog or screen-scrape the passphrase during unlock unless you have a trusted input path—which you don’t over KVM-IP.

    If the IPMI is bad, assume the box is owned. LUKS protects against lost disks, not malicious out-of-band management.

    Edit: Post reformatted with LLM to make it easier to understand.

  • ZizzyDizzyMCZizzyDizzyMC Hosting Provider
    edited February 1

    @somik said:

    @lowendmeow said:

    @skorous said:

    (if they offer kvm over ip)

    Or an interface has IPMI (mis)configured.

    What is the attack vector on dedicated devices if they have bad IPMI or KVM over IP? Would that be interfering with my unencrypted boot partition to keylog luks password etc?

    If IPMI / BMC / KVM-over-IP is compromised or malicious, you should assume full physical-equivalent access to the machine.

    That includes:

    • Pre-boot control: IPMI sits below the OS. A malicious BMC can tamper with the boot process before your kernel or initramfs ever runs.
    • LUKS password capture: KVM-over-IP literally sees (and can inject) keyboard input. Capturing the LUKS passphrase at boot is trivial.
    • Boot chain modification:

      • Modify the unencrypted /boot partition (kernel, initramfs, cmdline)
      • Inject a malicious initramfs that exfiltrates keys after unlock
      • Persist rootkits that survive reboots
    • Virtual media attacks: IPMI can mount “virtual CD/USB” and boot arbitrary images without you noticing.

    • DMA-style attacks: Some BMCs can read/write system memory directly.
    • Firmware persistence: A compromised BMC can re-infect the host OS even after a clean reinstall.

    Even if /boot is encrypted (or Secure Boot is enabled), a malicious BMC can still keylog or screen-scrape the passphrase during unlock unless you have a trusted input path—which you don’t over KVM-IP.

    If the IPMI is bad, assume the box is owned. LUKS protects against lost disks, not malicious out-of-band management.

    Edit: Post reformatted with LLM to make it easier to understand.

    Had someone once ask why I left IPMI unplugged.
    There's your answer.

    Thanked by (1)skorous
  • @ZizzyDizzyMC said:
    Had someone once ask why I left IPMI unplugged.
    There's your answer.

    I used to go for dedicated servers with IPMI. After I learned of the security issues, I either go for servers without IPMI or for servers where they provide you IPMI on "request", meaning they have to physically plug the ipmi into your server for it to work.

    Now a days, almost all the hosts provide IPMI even on the lower end servers. If you dont think they do, ask this question. Does your provider allow automated OS reinstallation or recovery mode or shell access for your dedi from the control panel? That's just a overlay on the existing IPMI layer...

    If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
    It’s OK if you disagree with me. I can’t force you to be right!

  • ..just get a box where IPMI is so outdated you need to have IE5 and ActiveX plugins available. It'll take them longer to setup an attack vector to connect to it that they'll just give up. The only thing worse than the ActiveX IPMI are the 'needs a specific version of java and ancient encryption enabled to work".

    Even my cheap NOCIX box is IPMI on request.

    Thanked by (1)quicksilver03

    My pronouns are like/subscribe.

  • @WSS said:
    ..just get a box where IPMI is so outdated you need to have IE5 and ActiveX plugins available. It'll take them longer to setup an attack vector to connect to it that they'll just give up. The only thing worse than the ActiveX IPMI are the 'needs a specific version of java and ancient encryption enabled to work".

    Even my cheap NOCIX box is IPMI on request.

    LOL just went through that ancient java one on my OVH dedi. The IE5 and ActiveX blows my mind. Funny that these older ones actually give you more safety. I know it was a pain in the ass for me so an attacker would give up quick

  • @lowendmeow said: The IE5 and ActiveX blows my mind.

    As memory serves, old Dell IDrac, and possibly really old SuperMicro. Can't remember what I had on my Core2Duo, but it was some proprietary shit that was hard to make work in 2015, little alone now.

    My pronouns are like/subscribe.

  • skorousskorous OGSenpai

    @WSS said: old Dell IDrac

    Can confirm. Just got rid of my last one of those maybe a year ago I think.

    Thanked by (1)WSS
  • WSSWSS OG

    @skorous said:

    @WSS said: old Dell IDrac

    Can confirm. Just got rid of my last one of those maybe a year ago I think.

    Those were abysmal.

    Thanked by (1)skorous

    My pronouns are like/subscribe.

  • Re: the original encryption question: adding additional layers of local encryption may make your system different enough from the rest, that an attacker won't bother to get your data. But it's a weak insurance.

    The only foolproof way to store encrypted data on a VPS, is if the VPS itself can't decrypt it. For example a remote Borg backup, or another system where the encryption/decryption steps happen outside the VPS, and the VPS is simply used as dumb storage.

    @lowendmeow said:
    Funny that these older ones actually give you more safety. I know it was a pain in the ass for me so an attacker would give up quick

    Sorry, no it's not - that's terrible advice. Please don't intentinally use old, insecure software. Security by obscurity is not enough and you can't rely on it
    https://en.wikipedia.org/wiki/Security_through_obscurity

    Thanked by (1)lowendmeow
  • AuroraZeroAuroraZero Retired

    Going to give everyone a piece of advice. If you piss someone off enough or they become infatuated with you and they have a modicum of skills they will work until they get what they want. It will make no difference to them the time involved nor will it matter what defenses they hit.

    The human factor is the weakest point of any setup and if they have to they will exploit that either digitally or in the real world. There are very few people that take security seriously enough to avoid the issue entirely.

    P.S. Dell IDRAC can go blow a whale in mating season with no condom.

  • @IAmNix said:
    Re: the original encryption question: adding additional layers of local encryption may make your system different enough from the rest, that an attacker won't bother to get your data. But it's a weak insurance.

    The only foolproof way to store encrypted data on a VPS, is if the VPS itself can't decrypt it. For example a remote Borg backup, or another system where the encryption/decryption steps happen outside the VPS, and the VPS is simply used as dumb storage.

    Yes I understand I had a brainfart there and realized systemd-homed is just LUKS under the hood as well and the key would also be in the memory dump, although you're right that it would make it more tiresome forensically. I do really like systemd-homed on my portable Linux devices though especially when traveling internationally. The other option would be to be on a VPS that has encrypted ram correct? I think I heard @MannDude supports that? I'm hoping we'll see more providers do so

    @lowendmeow said:
    Funny that these older ones actually give you more safety. I know it was a pain in the ass for me so an attacker would give up quick

    Sorry, no it's not - that's terrible advice. Please don't intentinally use old, insecure software. Security by obscurity is not enough and you can't rely on it
    https://en.wikipedia.org/wiki/Security_through_obscurity

    Yes sorry I was joking a bit with that reply not true safety but making it more obnoxious or obsolete for attackers. Like just because Win 95 may not be hit by the latest viruses not a reason to run it :D

  • neelcneelc Hosting ProviderServices Provider

    @ZizzyDizzyMC said:
    Had someone once ask why I left IPMI unplugged.
    There's your answer.

    For me, I leave IPMI on a separate VLAN, with internal IPs. You need a VPN to access them.

    My previous VPS host venture had IPMI on public IPv4, due to lacking a core router (to segment networks).

Sign In or Register to comment.