Sure I'll venture a guess.
Virmach has remote hands checking and/or trying to recover a broken drive. I don't think the node is down, only some VMs are down on TYOC027. We were just lucky enough to be on a drive that has issues. If the drive can't be recovered, it will be replaced and restored from backups. This could take a few more days or a week depending on DC hands workload and what needs to be done.
When the Los Angeles migrations are finished, I expect that VirMach may have time to deal with this, but right now it is probably not the top priority as relatively few VMs are affected. I know this probably doesn't make you feel any better about this downtime, but that is my guess. If you are really in a bind, and have your own backups to restore, PM me with what size VM you have and I'll see if there is something I can lend you in the meantime.
Of course maybe you'll get lucky and because I said it could take a week, it will be up in a hour just to show how wrong I am.
TYO-C027 has not been recovered so far, is anyone following up and dealing with it? @VirMach@FrankZ
Just wanted to say thank you! I've got my VM back up that was on LAXA0007.
I know it isn't a priority, and you are busy with many other things; With QPS in the loop will we be possibly getting IPv6?
There’s a bullet point somewhere mentioning IPv6 and Los Angeles.
@FrankZ said:
It still makes me chuckle when people tag me along with VirMach for a service related issue.
You guys should know that I do not, and never have, worked for VirMach.
I am just a long time customer.
@geo said:
Damn. LAXA019 was online for a while then offline.
Looks like I'll be waiting along with you on LAXA019.
This one needed to be physically unplugged and plugged back in, no idea why. Probably related to BMC/IPMI on these boards and particular scenarios where everything randomly breaks.
LAXA006 similar issue but needed BIOS configurations set again (but unrelated to CMOS battery or anything like that, just a feature.) Still not 100% sure.
LAXA014, LAXA015, LAXA016 related to SolusVM bug and abuse (IP stealing, probably unintentional.)
141.11.95.0/24 mentioned earlier related to me being dumb and missing a few blocks on initial announcement request (I copied the same row twice on a spreadsheet, skill/hand-eye coordination issue.)
Hi, I see "LAX1Z013 - Complete" in February 26th's Network Status page, not metioned in February 27th's though. Does it mean the service is back to normal? Mine is not. It went back online at about 05:30 (UTC +8), then turned to unpingable at about 13:57 (UTC +8). It can be accessible via VNC, but can't connect to the internet.
I tried Troubleshooter, it shows :
Main IP pings: false
Node Online: true
Service online: online
Operating System:
Service Status:Active
Registration Date:2021-09-23
Then I tried "Reconfigure Networking", it shows:
An Error Occurred!
Networking could not be reconfigured. Unknown operating system
@dudu said: Hi, I see "LAX1Z013 - Complete" in February 26th's Network Status page, not metioned in February 27th's
May be delayed.
Affecting System - LAXA028,LAX1Z013,LAXA005,LAXA011,LAXA024,LAXA031,LAX1Z019
02/27/2025 01:20 Last Updated 02/27/2025 10:40
LATEST UPDATE — LAXA031 is being worked on
The following servers are going to have thermal paste re-applied as we've noticed higher CPU temperatures. This is unrelated to the current maintenance window but we are going to complete this necessary maintenance to avoid having to schedule another window in the future.
LAXA028 (critical) -- COMPLETE LAX1Z013 -- Delayed
LAXA005 -- Delayed
LAXA011 -- Delayed
LAXA012 -- Complete
LAXA024 -- Planned soon
LAXA031 (also NVMe thermals checked) -- Delayed
LAX1Z019 (not CPU, but NVMe thermals checked) -- Delayed
If listed as "Delayed" we may defer this until a later date.
The following servers may be brought offline to add an additional NVMe SSD, and some customers may be moved to the new NVMe (the second portion will not have any additional downtime)
LAXA022
LAX1Z015
LAXA014
LAX1Z017
P.S. LAXA019 has been online and working fine for @geo and I for the past three hours.
Thank you VirMach.
Okay I’m not doing any further maintenance outside of maybe reseating PCIe, I had to end up doing a board swap for LAXA031 because of a BIOS update. In other news LAXA031 has been upgraded from a 5950X to a 3900X. I could put the 5950X back in but my new no maintenance policy prohibits it as I’d like to have the option to leave the facility today.
@dudu said:
Hi, I see "LAX1Z013 - Complete" in February 26th's Network Status page, not metioned in February 27th's though. Does it mean the service is back to normal? Mine is not. It went back online at about 05:30 (UTC +8), then turned to unpingable at about 13:57 (UTC +8). It can be accessible via VNC, but can't connect to the internet.
I tried Troubleshooter, it shows :
Main IP pings: false
Node Online: true
Service online: online
Operating System:
Service Status:Active
Registration Date:2021-09-23
Then I tried "Reconfigure Networking", it shows:
An Error Occurred!
Networking could not be reconfigured. Unknown operating system
Second error means you have to do it manually. Open a ticket but use the troubleshooter option if it’s available aka don’t do priority, that’s being spammed right now. Don’t mention maintenance or LAX in the title. Actually just give me the ticket ID.
If it’s not internal to the VPS then I’m aware of a few secondary blocks that need fixing.
LAXA014 going down isn't part of the maintenance, it's an actual outage. Let's see how quick I can resolve it in this rare scenario where I'm right there as it happens.
@VirMach said: In other news LAXA031 has been upgraded from a 5950X to a 3900X.
FYI - I can access VPS on LAXA031 via VNC, VPS is running.
No network. can't ping the gateway .129, can't ping 1.1.1.1. "Destination Host unreachable".
Reconfigure network does not solve. Manual check of network inside VPS looks good. BGP has not changed since the 24th.
@VirMach said: In other news LAXA031 has been upgraded from a 5950X to a 3900X.
FYI - I can access VPS on LAXA031 via VNC, VPS is running.
No network. can't ping the gateway .129, can't ping 1.1.1.1. "Destination Host unreachable".
Reconfigure network does not solve. Manual check of network inside VPS looks good. BGP has not changed since the 24th.
Yeah that sounds like the “some secondary subnets” scenario since you can access VNC. I’ll fix those remotely. It’s possible a /24 or /23 left over also got missed and not noticed as I’m focusing on getting remote access to everything and making plenty of other mistakes.
Okay LAXA014 has been fixed for a while but it’s back to being a network issue, I’ll hang a bit longer after bringing that one back up (hopefully for the last time) to see if anything else drops off then we should be good.
@VirMach said:
Okay everything should be good (enough to leave and deal with remotely)
Great!
It's time to start working on the offers for users affected by the outage: huge compensation, free configuration upgrades, free migrations, endless ipv6 and a coupon.
@VirMach said:
Okay everything should be good (enough to leave and deal with remotely)
Great!
It's time to start working on the offers for users affected by the outage: huge compensation, free configuration upgrades, free migrations, endless ipv6 and a coupon.
Best I can do is offer a free migration to an E5 in my bedroom. I get to power it down if it gets too loud, that’s the AUP.
I have been trying to turn on one of my VPSes in LAX1Z014 but it only shows "Powered off" after pressing the "On" button. I already did the troubleshoot steps (there's no ISO attached to the VPS, changed the boot order, rescue mode does not work, connecting to the VNC only shows "authentication failed", even changing the VNC password and force power off and on again does not let me connect to the VNC). Should I submit a non-emergency ticket or I wait a little bit longer?
@ricANNArdo said:
I have been trying to turn on one of my VPSes in LAX1Z014 but it only shows "Powered off" after pressing the "On" button. I already did the troubleshoot steps (there's no ISO attached to the VPS, changed the boot order, rescue mode does not work, connecting to the VNC only shows "authentication failed", even changing the VNC password and force power off and on again does not let me connect to the VNC). Should I submit a non-emergency ticket or I wait a little bit longer?
...nevermind, I can't even make a ticket:
Oops!
Something went wrong and we couldn't process your request.
Please go back to the previous page and try again.
Error: Call to undefined function ping() in [slash][redacted][slash]customTicketsAddon[slash]hooks.php
@ricANNArdo said:
I have been trying to turn on one of my VPSes in LAX1Z014 but it only shows "Powered off" after pressing the "On" button. I already did the troubleshoot steps (there's no ISO attached to the VPS, changed the boot order, rescue mode does not work, connecting to the VNC only shows "authentication failed", even changing the VNC password and force power off and on again does not let me connect to the VNC). Should I submit a non-emergency ticket or I wait a little bit longer?
...nevermind, I can't even make a ticket:
Oops!
Something went wrong and we couldn't process your request.
Please go back to the previous page and try again.
Error: Call to undefined function ping() in [slash][redacted][slash]customTicketsAddon[slash]hooks.php
Someone was supposed to fix that ticket error when it got reported earlier, hmm. Okay I know what’s wrong based on that though no need for a ticket. Working on it now
@sh97 said:
anyway to know what node my VM was on (its now down)
The IP is 141.11.92.xxx
Fixing all these control and ticket issues. I've also partially coded something out for what you're requesting, it's basically a replacement page for the current single message "something might be wrong" page. Not ready yet. So to answer your question, no. Also not that it'd help you at this point anyway.
@VirMach said: Second error means you have to do it manually
I just want to add a huge disclaimer on what I said here, I was just saying that in cases where the network reconfigure tool fails, you have to do it manually. I shouldn't have focused on that at all, and should have instead said there is almost no reason to reconfigure networking within your VPS as a result of the migration.
I'm exhausted so muscle memory kicked in and I've been looking at tickets where a lot of people are doing this, and I even did it myself, definitely don't do it. Nothing good will come of it. I'm going to add that to the status page in a final update.
Reason being is if it worked before, it'll work after once we fix any remaining issues on our end, and if you manually change things around, it's not going to work once we fix it, because for it to work, it needs to be how it used to be when it worked a few days ago. In the rare case that something has to change, you'll get an email.
@VirMach said: Reason being is if it worked before, it'll work after once we fix any remaining issues on our end, and if you manually change things around, it's not going to work once we fix it, because for it to work, it needs to be how it used to be when it worked a few days ago. In the rare case that something has to change, you'll get an email
Hello, I want to know how to download the data of my server after my service got suspended?
I have created a ticket to appeal the suspension but got no response at all for one week. My purpose is not to appeal the suspension here but to find out a way to download my data. It would be great if the virmach staff can answer my ticket (Ticket #238520) after seeing this post.
My service is not due yet. I just want to get my data back.
@tulipyun said: Although I can access the LAX VPS, hetrixtool keeps reporting offline warnings from all over the globe, should this be normal at this point?
@tulipyun said: Although I can access the LAX VPS, hetrixtool keeps reporting offline warnings from all over the globe, should this be normal at this point?
@tulipyun said:
Although I can access the LAX VPS, hetrixtool keeps reporting offline warnings from all over the globe, should this be normal at this point?
Adjusting some configurations on the switch to address it.
Nothing to do with BGP or the servers or anything else in this case, I just fixed the remaining network issues too quickly and it cascaded into this.
@nullgradient said:
Hello, I want to know how to download the data of my server after my service got suspended?
I have created a ticket to appeal the suspension but got no response at all for one week. My purpose is not to appeal the suspension here but to find out a way to download my data. It would be great if the virmach staff can answer my ticket (Ticket #238520) after seeing this post.
My service is not due yet. I just want to get my data back.
Operate as if the server is on fire: recover from backups.
Any data that lacks backups is unimportant data that you can live without.
We accept Karma donations for the last flan. 🍮 affbrr
@nullgradient said:
Hello, I want to know how to download the data of my server after my service got suspended?
I have created a ticket to appeal the suspension but got no response at all for one week. My purpose is not to appeal the suspension here but to find out a way to download my data. It would be great if the virmach staff can answer my ticket (Ticket #238520) after seeing this post.
My service is not due yet. I just want to get my data back.
Operate as if the server is on fire: recover from backups.
Any data that lacks backups is unimportant data that you can live without.
You are right. But you still don't want to randomly lose data just because they are not important enough, right?
My service is not cancelled or overdue, so the data are still supposed be there.
The next due date of my service is months away, I just want to do something about it before it's too late because I don't know how long it will be before my ticket are reviewed.
@nullgradient said:
Hello, I want to know how to download the data of my server after my service got suspended?
I have created a ticket to appeal the suspension but got no response at all for one week. My purpose is not to appeal the suspension here but to find out a way to download my data. It would be great if the virmach staff can answer my ticket (Ticket #238520) after seeing this post.
My service is not due yet. I just want to get my data back.
Operate as if the server is on fire: recover from backups.
Any data that lacks backups is unimportant data that you can live without.
You are right. But you still don't want to randomly lose data just because they are not important enough, right?
My service is not cancelled or overdue, so the data are still supposed be there.
The next due date of my service is months away, I just want to do something about it before it's too late because I don't know how long it will be before my ticket are reviewed.
I think it's late already. Next time configure a backup strategy ahead of time.
Comments
Well that's sounds good idea!
Good morning.
I'm at the beach today but there is no flan available in the hotel.
Also all my Virmach servers are online.
good day huh?
Virmach Deals
There’s a bullet point somewhere mentioning IPv6 and Los Angeles.
This one needed to be physically unplugged and plugged back in, no idea why. Probably related to BMC/IPMI on these boards and particular scenarios where everything randomly breaks.
LAXA006 similar issue but needed BIOS configurations set again (but unrelated to CMOS battery or anything like that, just a feature.) Still not 100% sure.
LAXA014, LAXA015, LAXA016 related to SolusVM bug and abuse (IP stealing, probably unintentional.)
141.11.95.0/24 mentioned earlier related to me being dumb and missing a few blocks on initial announcement request (I copied the same row twice on a spreadsheet, skill/hand-eye coordination issue.)
Monitor name: box5 VirMach LAX
Root cause: Host Is Unreachable
Incident started at: 2025-02-25 16:58:35
Resolved at: 2025-02-27 16:51:16
Duration: 47 hours and 52 minutes
Waiting SLA credits.
A refund equivalent of 1-month of service cost sounds reasonable.
$0/month
We accept Karma donations for the last flan. 🍮 affbrr
Hi, I see "LAX1Z013 - Complete" in February 26th's Network Status page, not metioned in February 27th's though. Does it mean the service is back to normal? Mine is not. It went back online at about 05:30 (UTC +8), then turned to unpingable at about 13:57 (UTC +8). It can be accessible via VNC, but can't connect to the internet.
I tried Troubleshooter, it shows :
Then I tried "Reconfigure Networking", it shows:
What should I do now? Do you aware of that?
cc @VirMach
3 out of 6 is online.
May be delayed.
P.S. LAXA019 has been online and working fine for @geo and I for the past three hours.
Thank you VirMach.
I've warned you before about posting in offer threads.
You need to take a 24 hr timeout.
Go to the movies, or maybe bake a cake.
See you tomorrow.
The latest regarding these nodes from the status page:
Okay I’m not doing any further maintenance outside of maybe reseating PCIe, I had to end up doing a board swap for LAXA031 because of a BIOS update. In other news LAXA031 has been upgraded from a 5950X to a 3900X. I could put the 5950X back in but my new no maintenance policy prohibits it as I’d like to have the option to leave the facility today.
Second error means you have to do it manually. Open a ticket but use the troubleshooter option if it’s available aka don’t do priority, that’s being spammed right now. Don’t mention maintenance or LAX in the title. Actually just give me the ticket ID.
If it’s not internal to the VPS then I’m aware of a few secondary blocks that need fixing.
Downgraded you mean?
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
LAXA014 going down isn't part of the maintenance, it's an actual outage. Let's see how quick I can resolve it in this rare scenario where I'm right there as it happens.
Def an upgrade.
We used to live in a 3-story.
Nowadays we live in a studio apartment.
Cleaning is so much easier.
We accept Karma donations for the last flan. 🍮 affbrr
FYI - I can access VPS on LAXA031 via VNC, VPS is running.
No network. can't ping the gateway .129, can't ping 1.1.1.1. "Destination Host unreachable".
Reconfigure network does not solve. Manual check of network inside VPS looks good. BGP has not changed since the 24th.
Yeah that sounds like the “some secondary subnets” scenario since you can access VNC. I’ll fix those remotely. It’s possible a /24 or /23 left over also got missed and not noticed as I’m focusing on getting remote access to everything and making plenty of other mistakes.
Almost everything being rebooted a final time due to motherboard bug, in case anyone’s wondering.
This one’s my favorite bug it breaks IPMI and looks no different than it being set correctly
Okay LAXA014 has been fixed for a while but it’s back to being a network issue, I’ll hang a bit longer after bringing that one back up (hopefully for the last time) to see if anything else drops off then we should be good.
Okay everything should be good (enough to leave and deal with remotely)
Great!
It's time to start working on the offers for users affected by the outage: huge compensation, free configuration upgrades, free migrations, endless ipv6 and a coupon.
Best I can do is offer a free migration to an E5 in my bedroom. I get to power it down if it gets too loud, that’s the AUP.
Build some custom rack slots under your mattress and you can get rid of your electric blanket...
QuickPacket - Dedicated Servers in Ashburn, Los Angeles, Chicago
I have been trying to turn on one of my VPSes in LAX1Z014 but it only shows "Powered off" after pressing the "On" button. I already did the troubleshoot steps (there's no ISO attached to the VPS, changed the boot order, rescue mode does not work, connecting to the VNC only shows "authentication failed", even changing the VNC password and force power off and on again does not let me connect to the VNC). Should I submit a non-emergency ticket or I wait a little bit longer?
You probably saw me somewhere...
@FrankZ Thanks for fixing everything!!!1111 Best support evr.
Also thanks @VirMach & @qps for helping boss man fix everything.
@virmach where is the 1200GB NVMe server?
You know what would be better? A server with a couple of 2TB NVMe drives. The nearest to Dallas, the better.
Debían, thx
Since when? @FrankZ
Virmach Deals
...nevermind, I can't even make a ticket:
You probably saw me somewhere...
Someone was supposed to fix that ticket error when it got reported earlier, hmm. Okay I know what’s wrong based on that though no need for a ticket. Working on it now
anyway to know what node my VM was on (its now down)
The IP is 141.11.92.xxx
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
Fixing all these control and ticket issues. I've also partially coded something out for what you're requesting, it's basically a replacement page for the current single message "something might be wrong" page. Not ready yet. So to answer your question, no. Also not that it'd help you at this point anyway.
I'm not a fan of the spam either, but just to point out, this isn't an offer thread.
I'm assuming it got moved here from the other thread.
You can directly go to the CP from service list
Virmach Deals
Got it, its on LAXA006
Thanks!
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
I just want to add a huge disclaimer on what I said here, I was just saying that in cases where the network reconfigure tool fails, you have to do it manually. I shouldn't have focused on that at all, and should have instead said there is almost no reason to reconfigure networking within your VPS as a result of the migration.
I'm exhausted so muscle memory kicked in and I've been looking at tickets where a lot of people are doing this, and I even did it myself, definitely don't do it. Nothing good will come of it. I'm going to add that to the status page in a final update.
Reason being is if it worked before, it'll work after once we fix any remaining issues on our end, and if you manually change things around, it's not going to work once we fix it, because for it to work, it needs to be how it used to be when it worked a few days ago. In the rare case that something has to change, you'll get an email.
Understood
Virmach Deals
My VM on LAXA006 came back up. Gotta say - PREM NETWORK.
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
Hello, I want to know how to download the data of my server after my service got suspended?
I have created a ticket to appeal the suspension but got no response at all for one week. My purpose is not to appeal the suspension here but to find out a way to download my data. It would be great if the virmach staff can answer my ticket (Ticket #238520) after seeing this post.
My service is not due yet. I just want to get my data back.
Although I can access the LAX VPS, hetrixtool keeps reporting offline warnings from all over the globe, should this be normal at this point?
Reason might be this: https://bgp.he.net/ip/45.143.11.1 (currently announced by 2 providers)
in my case i can reach my vps on LAX1Z021 via vnc, but not from internet, also the vps cant reach anything outside.
Thanks for the answer, that makes sense!
Adjusting some configurations on the switch to address it.
Nothing to do with BGP or the servers or anything else in this case, I just fixed the remaining network issues too quickly and it cascaded into this.
Looks like something went wrong with one of my VPS on node "RYZE.LAX-A019.VMS", after the migration.
Anyone else has a similar issues?
Someone is playing flip-flop game
Virmach Deals
Operate as if the server is on fire: recover from backups.
Any data that lacks backups is unimportant data that you can live without.
We accept Karma donations for the last flan. 🍮 affbrr
You are right. But you still don't want to randomly lose data just because they are not important enough, right?
My service is not cancelled or overdue, so the data are still supposed be there.
The next due date of my service is months away, I just want to do something about it before it's too late because I don't know how long it will be before my ticket are reviewed.
I think it's late already. Next time configure a backup strategy ahead of time.