@VirMach Solus re-config. network does the job improperly for vps setup using iso. Please just let it be an option for whoever needs. My vps are all OK now. Please don't touch them
@tuc said: @VirMach Solus re-config. network does the job improperly for vps setup using iso. Please just let it be an option for whoever needs. My vps are all OK now. Please don't touch them
I had that problem in the past, since Centos likes to do the ens3 thing (which RHEL thinks is a better way). However, Solus/Virmach's custom scripts seem to want to use eth0, eth0:0, etc. and screw things up. Now if I do a Centos install I change the network naming to eth0 specifically to try to mitigate that.
@VirMach said:
Okay, we're going to stop doing the automatic changes. We'll just let people change it themselves, and then proceed with automatic changes on the 29th. I've skimmed through the message in the same way anyone would read a wall of text and it definitely sounds like it would be changing on the 29th. We'll just update the page with when each node is available to change, and won't change the main IPv4 until that day.
I'm going to focus on trying to reconfigure any that were hanged and put a notice where people can contact us for their old address if desired in this case, because another 30 hours of adding the old IPv4 back and trying to make sure people get the exact old address without it breaking anything else sounds risky for no reason.
Yeah. That's the way I read it. Thought I had until 29th to change IPs - ideally that would have been during a low usage period. So this was a surprise. Once the new IP showed I lost all network connectivity. The weird thing was that the old IP was still in the ifcfg-eth0 file - so it didn't get changed automatically, but was also dead. I changed it manually (rebooted, etc.) but IP a still showed the same old IP which no longer worked. There are a bunch of files in /etc/sysconfig/network-scripts that I had never seen before so I renamed anything that looked like it could be read as a config file, and that seemed to have done the trick. Finally things seem to be working with the new IP so I hope things don't get reverted.
@tuc said: My vps are all OK now. Please don't touch them
@SeaScoot said: Finally things seem to be working with the new IP so I hope things don't get reverted.
I can't promise that, unfortunately, but we do try to avoid touching it unless necessary. In the case of these that got changed early, I've kept a list so we'll likely not do anything on September 29th for them, but reconfigurations will run today most likely for them if I can get it stop having issues.
Assuming it works correctly and your server pings, it'll skip over it. Otherwise if you have firewall blocking that, it'll reconfigure it. The script is more complicated than that and it also skips over some other scenarios as well to try to reduce impact. But it also breaks because it's the worst operation ever in terms of SolusVM API, so what may end up happening sometimes is if it fails really bad then it requires manual work and there's no way we can go one by one checking them before reconfiguring in that scenario and it'll just have to be mass reconfigured. Because if we don't we'll get a thousand tickets over the next week. This way we only get several hundred
Advanced probing feature by SolusVM. I wonder what it actually does... it wouldn't be very difficult for it to do it properly instead of breaking it.
@SeaScoot said: The weird thing was that the old IP was still in the ifcfg-eth0 file - so it didn't get changed automatically, but was also dead. I changed it manually (rebooted, etc.) but IP a still showed the same old IP which no longer worked. There are a bunch of files in /etc/sysconfig/network-scripts that I had never seen before so I renamed anything that looked like it could be read as a config file, and that seemed to have done the trick
Yeah, I really don't know what change they made, I read through all the changelogs. Supposedly the thing they "fixed"... was "fixed" in 2021 (it wasn't.)
I still suspect it's doing a halfway reconfigure very poorly. Because just doing the network update I mentioned shouldn't overload the node like that if it's done too quickly, and I noticed that whatever it was doing was causing some services to come online, which shouldn't be possible unless it's doing some reconfiguration. I haven't had time to dedicate to trying to figure out what the hell they did but it would have obviously been great if I did.
LAX2Z018 node has received an email about the change in IP address, but it has not provided the actual IP address.
Dear VirMach Customer,
Quick Version: Your service's IP address will be changed on September 29th. It is not ready yet, but we wanted to give you advanced notice that it will be ****.
@tbnuser said:
The recovery wasn't too abysmal thankfully.
Just need to wait for the network to settle down now though, getting anywhere between 600-1000ms on the first hop running mtr to google
Oof, yeah definitely need the network to settle down in LA. Right now it's bouncing between normal 10-20ms latency, spiking up to 250-300ms latency, or rocketing up to 1000-3000ms latency with random timeouts. It'll be fine for 10 minutes, then a couple mins of barely reachable. It was pretty solid last night into the early morning, with none of those problems, I was thinking how smoothly switching IPs had gone for me, this is probably my fault for jinxing it haha
@bakageta said: It was pretty solid last night into the early morning, with none of those problems
What nodes? That's important. Last night into early morning it was smooth because we made some changes, now I just need to know which nodes had problems at what time so I can go through the monitors.
@bakageta said: It was pretty solid last night into the early morning, with none of those problems
What nodes? That's important. Last night into early morning it was smooth because we made some changes, now I just need to know which nodes had problems at what time so I can go through the monitors.
Oh right, that might help eh? I noticed it on LAXA023, taking a quick look at the rest of mine and I'm not seeing any issues on any other nodes personally.
(edit)
And this is why I didn't mention it in the first place, because it seems to have calmed down now and is behaving normally.
@VirMach Please check that the computer being migrated from Denver 02 to LAXA005 has power. The new IP is displayed, but the hard drive is not. The machine won't boot and can't connect to VNC.
@bakageta said: It was pretty solid last night into the early morning, with none of those problems
What nodes? That's important. Last night into early morning it was smooth because we made some changes, now I just need to know which nodes had problems at what time so I can go through the monitors.
Oh right, that might help eh? I noticed it on LAXA023, taking a quick look at the rest of mine and I'm not seeing any issues on any other nodes personally.
(edit)
And this is why I didn't mention it in the first place, because it seems to have calmed down now and is behaving normally.
I do see what you see around that time, maybe half an hour before your post. Unrelated to the first issue, it was just some network abuse (maybe?) Someone sent out like 500,000 PPS / 6Gbps for maybe an hour. I'll go spank him later.
It does make me concerned about the switch though, it should've handled that better. Adding it to my list.
If I ever end up going to the datacenter for the Seattle servers... I'll also plan out finally finalizing the cabinet in LAX. It was never meant to be on this switch for this long, maybe I'll take down a QFX5100 & EX4300 or something. I have EX4300's everywhere around the office, I think I bought a few too many.
@VirMach said:
If I ever end up going to the datacenter for the Seattle servers... I'll also plan out finally finalizing the cabinet in LAX. It was never meant to be on this switch for this long, maybe I'll take down a QFX5100 & EX4300 or something. I have EX4300's everywhere around the office, I think I bought a few too many.
The new IP that was already in effect dropped out.
After checking, the original correct ens3 in /etc/network/interfaces was rewritten to eth0, I manually changed it back to ens3 and the server came back online immediately.
It happened on servers LAXA009 and LAXA026 successively.
@FAT32 said:
Sometimes I always wonder why unfortunate things happen to good people. Although VirMach is doing a business here, but a lot of times I feel like it is a charity, things just broke randomly and VirMach still tried his best to keep things afloat.
When VirMach migrate away from CC, a lot of people say VirMach probably cannot survive with those insane deals (come on, $0.95/yr with IPv4? That cannot even cover the cost of IP), but yet they did. Now when DediPath encountered issue, VirMach still managed to resolve it so quickly and everything seems to be smooth on my side as a user.
Sure, there are times where support can be slow or poor, but based on my ~6-7 years of usage I don't really encounter major hiccups. Even with all the inflation, as far as I know VirMach never increased recurring price of any services. (Keep in mind the pricing are rock-bottom, literally unsustainable)
Thanks VirMach, just want to show my appreciation here after following probably thousand pages of forum posts (OGF/LES), hope that one day everything can back to normal and we can see VirMach 10th anniversary deals
I bought my first ever VPS from VirMach in 2017. Had only used shared hosting before that. Then I got introduced to insane VirMach deals on BF/Cyber Monday. Sometimes I think Sohail is a very rich guy and he just wants to keep us happy that is why he is still going.
LAX2Z019 update, kind of. The situation didn't improve at all, and at this point it's basically 0% chance the NVMe can be brought back to life. The capacitors flew off the PCIe riser so there was probably a voltage spike, and another capacitor on the NVMe looks pretty bad. Going to call it later today, and we'll get back up the portion unaffected and recreate the rest.
Chicago2 is routed properly now. Still waiting on Tampa.
@VirMach
My VPS on node "RYZE.LAX-A026.VMS" not received a new IP address. The SolusVM panel listing only one old: "47.87.153.XX" (ZenLayer / Alicloud(?)) IP address. Is this a mistake? or normal?
I ask it only because if I understood correctly, the new IP assigning process on all LAX VM's has been ended (Networks status page info). Sorry, if I understood something wrong. Thx for the answer!
@titus said: @VirMach
My VPS on node "RYZE.LAX-A026.VMS" not received a new IP address. The SolusVM panel listing only one old: "47.87.153.XX" (ZenLayer / Alicloud(?)) IP address. Is this a mistake? or normal?
I ask it only because if I understood correctly, the new IP assigning process on all LAX VM's has been ended (Networks status page info). Sorry, if I understood something wrong. Thx for the answer!
That's one of the nodes that ran out and need to be moved around, you'll get a separate notification and/or it'll be posted on network status page. We haven't got to that yet but trying to do it in a way where you still get some notice. So you're part of the lucky 2%~ of people affected by that. Doesn't go by node for that, but probably like a dozen people on some nodes, something like that.
Tampa new IPv4 ready. ⌛ You may manually change this to your main IPv4 address in the full VPS control panel early, or continue using the old IPv4 address until September 29th.
My IP was not pinging and I reconfigured yesterday so I went to reconfigure again.... now VNC not loading (Starting VNC handshake), Status Unknown, Operation Timed Out After 90001 Milliseconds With 0 Bytes Received. Will re-check back in like 20 minutes to see what have I done and maybe it's SolusVM feature. :-D
Few minutes later server rebooted, but no network access on new IP (-:
[Mod edit: Now that it is fixed, I have removed the screenshot showing Jab's IP addresses.]
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
@VirMach said:
LAX2Z019 update, kind of. The situation didn't improve at all, and at this point it's basically 0% chance the NVMe can be brought back to life. The capacitors flew off the PCIe riser so there was probably a voltage spike, and another capacitor on the NVMe looks pretty bad. Going to call it later today, and we'll get back up the portion unaffected and recreate the rest.
Chicago2 is routed properly now. Still waiting on Tampa.
@Virmach: For machines that used to have more than one IP address but are now only allocated one is this a ticket occasion? It's not an issue for me as I wasn't currently using it but wanted to make sure there wasn't an automatic process which was going to follow up if I did nothing.
@skorous said: @Virmach: For machines that used to have more than one IP address but are now only allocated one is this a ticket occasion? It's not an issue for me as I wasn't currently using it but wanted to make sure there wasn't an automatic process which was going to follow up if I did nothing.
That shouldn't have happened using our scripts, it specifically adds in same number you used to have but it's possible in a rare case if it ran out that you only got one, in which case yeah you'd contact us. Just wait until a little bit later if possible.
@skorous said: @Virmach: For machines that used to have more than one IP address but are now only allocated one is this a ticket occasion? It's not an issue for me as I wasn't currently using it but wanted to make sure there wasn't an automatic process which was going to follow up if I did nothing.
That shouldn't have happened using our scripts, it specifically adds in same number you used to have but it's possible in a rare case if it ran out that you only got one, in which case yeah you'd contact us. Just wait until a little bit later if possible.
Rats, hoped you hadn't noticed me. As I edited the original post, I'm just dumb today. Sorry to bother.
FFME002.VIRM.AC seems to died like 15 minutes ago.
No ping on virmach status thing, no ping on my VM, panel not loading details - timeout.
VNC says Failed to connect to server (code: 1011, reason: Failed to connect to downstream server)
Sad times -:(
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
@Jab said:
FFME002.VIRM.AC seems to died like 15 minutes ago.
No ping on virmach status thing, no ping on my VM, panel not loading details - timeout.
VNC says Failed to connect to server (code: 1011, reason: Failed to connect to downstream server)
Uptime 20 minutes. My VM is back and working. If that was disk that wasn't mine disk and I am very sorry for rest of you!
Thanks VirMach team.
It'll always have a bump up like that, when it reboots since it's booting up like probably 100+ entire operating systems. The actual crash is before that bump. It was at only 14 load, 49% CPU, 82GB active memory. I haven't looked into it yet but I figured I'd just reboot it since it didn't look like any major errors.
Okay I just finished racking Seattle replacement servers in LAX. Warning -- graphic photo, kind of... Just putting spoiler tags for anyone who doesn't want to see it, click at your own risk. I forgot to take an actual photo after I was done, but it's pretty funny this happened in like the first 5 minutes there.
Comments
@VirMach Solus re-config. network does the job improperly for vps setup using iso. Please just let it be an option for whoever needs. My vps are all OK now. Please don't touch them
Virmach Deals
thoroughly molests @tuc 's VPS
The Yeti has left the building.
reconfig sets to eth0
but im using ens3
I bench YABS 24/7/365 unless it's a leap year.
I had that problem in the past, since Centos likes to do the ens3 thing (which RHEL thinks is a better way). However, Solus/Virmach's custom scripts seem to want to use eth0, eth0:0, etc. and screw things up. Now if I do a Centos install I change the network naming to eth0 specifically to try to mitigate that.
Cloudcone | Crunchbits | Dartnode | Dedirock | Georgedatacenter | Gigahost | Hostbilby | Host-c | Hostdzire | Hostsailor | Namecrane | Nuyek | Realtoxmedia | Servarica | Terabit | Tnahosting | Virmach
Yeah. That's the way I read it. Thought I had until 29th to change IPs - ideally that would have been during a low usage period. So this was a surprise. Once the new IP showed I lost all network connectivity. The weird thing was that the old IP was still in the ifcfg-eth0 file - so it didn't get changed automatically, but was also dead. I changed it manually (rebooted, etc.) but IP a still showed the same old IP which no longer worked. There are a bunch of files in /etc/sysconfig/network-scripts that I had never seen before so I renamed anything that looked like it could be read as a config file, and that seemed to have done the trick. Finally things seem to be working with the new IP so I hope things don't get reverted.
I can't promise that, unfortunately, but we do try to avoid touching it unless necessary. In the case of these that got changed early, I've kept a list so we'll likely not do anything on September 29th for them, but reconfigurations will run today most likely for them if I can get it stop having issues.
Assuming it works correctly and your server pings, it'll skip over it. Otherwise if you have firewall blocking that, it'll reconfigure it. The script is more complicated than that and it also skips over some other scenarios as well to try to reduce impact. But it also breaks because it's the worst operation ever in terms of SolusVM API, so what may end up happening sometimes is if it fails really bad then it requires manual work and there's no way we can go one by one checking them before reconfiguring in that scenario and it'll just have to be mass reconfigured. Because if we don't we'll get a thousand tickets over the next week. This way we only get several hundred
Advanced probing feature by SolusVM. I wonder what it actually does... it wouldn't be very difficult for it to do it properly instead of breaking it.
Yeah, I really don't know what change they made, I read through all the changelogs. Supposedly the thing they "fixed"... was "fixed" in 2021 (it wasn't.)
I still suspect it's doing a halfway reconfigure very poorly. Because just doing the network update I mentioned shouldn't overload the node like that if it's done too quickly, and I noticed that whatever it was doing was causing some services to come online, which shouldn't be possible unless it's doing some reconfiguration. I haven't had time to dedicate to trying to figure out what the hell they did but it would have obviously been great if I did.
LAX2Z018 node has received an email about the change in IP address, but it has not provided the actual IP address.
Dear VirMach Customer,
Quick Version: Your service's IP address will be changed on September 29th. It is not ready yet, but we wanted to give you advanced notice that it will be ****.
@adef_1 - Look in SolusVM I expect it will show up there when it is assigned, it did for me.
@VirNach, on LAXA-005, is 66.59.xxx.xxx will be permanent or 141.11.xxx.xx?
I'm betting on 141.11.xxx.xx
The recovery wasn't too abysmal thankfully.
Just need to wait for the network to settle down now though, getting anywhere between 600-1000ms on the first hop running mtr to google
(changed) ✅ Tampa has new IPv4 assigned but ❌ it has routing issues, we are waiting on the datacenter. Do not switch to new IPv4 yet.
Seems like my complaining worked, this is nice middle ground I would say
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Oof, yeah definitely need the network to settle down in LA. Right now it's bouncing between normal 10-20ms latency, spiking up to 250-300ms latency, or rocketing up to 1000-3000ms latency with random timeouts. It'll be fine for 10 minutes, then a couple mins of barely reachable. It was pretty solid last night into the early morning, with none of those problems, I was thinking how smoothly switching IPs had gone for me, this is probably my fault for jinxing it haha
What nodes? That's important. Last night into early morning it was smooth because we made some changes, now I just need to know which nodes had problems at what time so I can go through the monitors.
Oh right, that might help eh? I noticed it on LAXA023, taking a quick look at the rest of mine and I'm not seeing any issues on any other nodes personally.
(edit)
And this is why I didn't mention it in the first place, because it seems to have calmed down now and is behaving normally.
Where are the Atlanta Club fans? :-|
@VirMach Please check that the computer being migrated from Denver 02 to LAXA005 has power. The new IP is displayed, but the hard drive is not. The machine won't boot and can't connect to VNC.
I do see what you see around that time, maybe half an hour before your post. Unrelated to the first issue, it was just some network abuse (maybe?) Someone sent out like 500,000 PPS / 6Gbps for maybe an hour. I'll go spank him later.
It does make me concerned about the switch though, it should've handled that better. Adding it to my list.
If I ever end up going to the datacenter for the Seattle servers... I'll also plan out finally finalizing the cabinet in LAX. It was never meant to be on this switch for this long, maybe I'll take down a QFX5100 & EX4300 or something. I have EX4300's everywhere around the office, I think I bought a few too many.
give us one EX4300, we are missing one. LOL
George Datacenter LLC
www.georgedatacenter.com
Owner Hardware
Looks like RYZE.LAX-A022.VMS is stable again, it was a bit on and off yesterday
The new IP that was already in effect dropped out.
After checking, the original correct ens3 in /etc/network/interfaces was rewritten to eth0, I manually changed it back to ens3 and the server came back online immediately.
It happened on servers LAXA009 and LAXA026 successively.
I am using Debian 11.
man, virmach update is better than sex
Fuck this 24/7 internet spew of trivia and celebrity bullshit.
It's all about the anticipation.
I bought my first ever VPS from VirMach in 2017. Had only used shared hosting before that. Then I got introduced to insane VirMach deals on BF/Cyber Monday. Sometimes I think Sohail is a very rich guy and he just wants to keep us happy that is why he is still going.
Powerful AMD Ryzen VPS (aff)
Hello, haven't you made a push ticket recently?The push I did last Thursday is not finished yet.
lol.
On sad note: Tampa IP addresses still not routable worky
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
From what I can tell, there are still problems with:
RYZE.LAX-A007.VMS
TPAZ004.VIRM.AC
Cloudcone | Crunchbits | Dartnode | Dedirock | Georgedatacenter | Gigahost | Hostbilby | Host-c | Hostdzire | Hostsailor | Namecrane | Nuyek | Realtoxmedia | Servarica | Terabit | Tnahosting | Virmach
MIAZ011 has both iPs usable, but it seems that they have different routing policy.
The destination is TunnelBroker.net Miami router.
The old IPv4 can reach there directly through FL-IX.
The new IPv4 takes a detour through Ashburn.
We accept Karma donations for the last flan. 🍮 affbrr
LAX2Z019 update, kind of. The situation didn't improve at all, and at this point it's basically 0% chance the NVMe can be brought back to life. The capacitors flew off the PCIe riser so there was probably a voltage spike, and another capacitor on the NVMe looks pretty bad. Going to call it later today, and we'll get back up the portion unaffected and recreate the rest.
Chicago2 is routed properly now. Still waiting on Tampa.
@VirMach
My VPS on node "RYZE.LAX-A026.VMS" not received a new IP address. The SolusVM panel listing only one old: "47.87.153.XX" (ZenLayer / Alicloud(?)) IP address. Is this a mistake? or normal?
I ask it only because if I understood correctly, the new IP assigning process on all LAX VM's has been ended (Networks status page info). Sorry, if I understood something wrong. Thx for the answer!
That's one of the nodes that ran out and need to be moved around, you'll get a separate notification and/or it'll be posted on network status page. We haven't got to that yet but trying to do it in a way where you still get some notice. So you're part of the lucky 2%~ of people affected by that. Doesn't go by node for that, but probably like a dozen people on some nodes, something like that.
My IP was not pinging and I reconfigured yesterday so I went to reconfigure again.... now VNC not loading (Starting VNC handshake), Status Unknown, Operation Timed Out After 90001 Milliseconds With 0 Bytes Received. Will re-check back in like 20 minutes to see what have I done and maybe it's SolusVM feature. :-D
Few minutes later server rebooted, but no network access on new IP (-:
[Mod edit: Now that it is fixed, I have removed the screenshot showing Jab's IP addresses.]
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
@Jab fixed
Can confirmed, worky worky, much many thanks!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
LMAO what drive + riser model was that
👍
@Virmach: For machines that used to have more than one IP address but are now only allocated one is this a ticket occasion? It's not an issue for me as I wasn't currently using it but wanted to make sure there wasn't an automatic process which was going to follow up if I did nothing.
EDIT: Nevermind, I'm an idiot.
That shouldn't have happened using our scripts, it specifically adds in same number you used to have but it's possible in a rare case if it ran out that you only got one, in which case yeah you'd contact us. Just wait until a little bit later if possible.
Rats, hoped you hadn't noticed me. As I edited the original post, I'm just dumb today. Sorry to bother.
FFME002.VIRM.AC seems to died like 15 minutes ago.
No ping on virmach status thing, no ping on my VM, panel not loading details - timeout.
VNC says Failed to connect to server (code: 1011, reason: Failed to connect to downstream server)
Sad times -:(
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Lalalalala can't hear you. Ughh
We back!
Uptime 20 minutes. My VM is back and working. If that was disk that wasn't mine disk and I am very sorry for rest of you!
Thanks VirMach team.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
It'll always have a bump up like that, when it reboots since it's booting up like probably 100+ entire operating systems. The actual crash is before that bump. It was at only 14 load, 49% CPU, 82GB active memory. I haven't looked into it yet but I figured I'd just reboot it since it didn't look like any major errors.
Hello, when will the push order start processing?
Push orders can take from days to weeks depending on what is going on.
Push orders are the lowest priority ticket.
Okay I just finished racking Seattle replacement servers in LAX. Warning -- graphic photo, kind of... Just putting spoiler tags for anyone who doesn't want to see it, click at your own risk. I forgot to take an actual photo after I was done, but it's pretty funny this happened in like the first 5 minutes there.
No uplink though, waiting on DC to diagnose.
vir vir is busy
i like vir vir when he is busy
vir vir is making our vir's more u know virrrr