Seems like I did not miss anything while I was away. /s
My Amsterdam and Frankfurt VMs are working fine now with the new IPs.
Just a quick note for Alma 8 users....
I found each of my three EU VMs stuck at a waiting on rc.local job during boot after the old IPv4 were removed. While using VNC and booting into Alma 8 rescue mode from the boot menu, not rescue mode in SolusVM, I was able to start the VMs normally and change what needed to be changed regarding IPv4 configurations.
EDIT: Spoke too soon, both Frankfurt VMs are unreachable again.
Oddly, Hetrix reports 47.87.196.xxx alternating between reachable and non-reachable, depending on which origin is attempting to hit it.
I guess it's a lingering BGP route?
Yeah, seems like something (or someone) shut down everything I had in Frankfurt/Amsterdam and I had to manually click BOOT in panel. Is this IP renumbering script going crazy?
Agent hasn't sent any data in the past: 16 minutes
Welp, there goes uptime
@erk said:
Oddly, Hetrix reports 47.87.196.xxx alternating between reachable and non-reachable, depending on which origin is attempting to hit it.
I guess it's a lingering BGP route?
Yep, they didn't even pull it correctly. Usually how it works is they would make de-announcement requests. They just unsigned the records, didn't pull it from ARIN. Which makes it easier to bring it back so whatever.
@Jab said: all new IPs just went down, lmao/panic.
@FrankZ said: EDIT: Spoke too soon, both Frankfurt VMs are unreachable again.
Frankfurt got rebooted. Amsterdam should still be in the same state as it was a few hours ago.
@erk said:
Oddly, Hetrix reports 47.87.196.xxx alternating between reachable and non-reachable, depending on which origin is attempting to hit it.
I guess it's a lingering BGP route?
Yep, they didn't even pull it correctly. Usually how it works is they would make de-announcement requests. They just unsigned the records, didn't pull it from ARIN. Which makes it easier to bring it back so whatever.
@Jab said: all new IPs just went down, lmao/panic.
@FrankZ said: EDIT: Spoke too soon, both Frankfurt VMs are unreachable again.
Frankfurt got rebooted. Amsterdam should still be in the same state as it was a few hours ago.
Yea, it's weird... I have a 47.87.141.xxx server in LA running a database and my web server (different provider) has had no issues connecting to it all day. I can also SSH into all my 47.87.xxx.xxx US VPS's. But none of them can access my web server.
@Jab said: Yeah, seems like something (or someone) shut down everything I had in Frankfurt/Amsterdam and I had to manually click BOOT in panel. Is this IP renumbering script going crazy?
@Jab said: I guess this is new entry in Network status, but this needs timestamps in some known timezone, @VirMach please next time - put timestamps on updates.
Vague on purpose as I'm not the one doing it.
I have someone helping out who's more interested in fixing everything than being graceful. It most likely means SolusVM is not being cooperative so he's instead using a hammer on the servers. We had to do it in a very non-SolusVM API way for it not to take half a day (which is funny since, you know... but that just means doing it the other way would've taken two.) Amsterdam didn't get rebooted, but VMs might be receiving additional work.
Frankfurt finally got rebooted, it got delayed until now.
Just a quick note for Alma 8 users....
I found each of my three EU VMs stuck at a waiting on rc.local job during boot after the old IPv4 were removed. While using VNC and booting into Alma 8 rescue mode from the boot menu, not rescue mode in SolusVM, I was able to start the VMs normally and change what needed to be changed regarding IPv4 configurations.
I put most of my VMs on DHCP back during the first wave of re-IP's. I just click Reboot and * poof * I've got the new IP.
@soulchief said: Yea, it's weird... I have a 47.87.141.xxx server in LA running a database and my web server (different provider) has had no issues connecting to it all day. I can also SSH into all my 47.87.xxx.xxx US VPS's. But none of them can access my web server.
I'm actually so happy it happened this way. Sure it's annoying and confusing but at least we only got like under 1,000 tickets instead of probably 2K+
@rockinmusicgv said:
NYCB009 has been down for about a day and seems to be on a different IP block. Is there any indication when that node might be restored?
Yeah, it didn't come back long enough for us to be able to complete the planned emergency migration. It's difficult to get back online too, we've been working on that for a long time. Still speaking with RAID controller manufacturer but now it seems like there's a whole set of other issues.
I don't have any more information at the time.
Does this mean I should be expecting an extended outage?
I tried to connect to my web server from my local location, timed out.
Connected to an VPN in NY, tried again, it worked!
My VPS is at the node SEAZ007, maybe it's something related to route propagation, maybe the previous IPs are being reestablished?
The problem looks like solved in Germany location (FFME003) with the new IP address. My VM accessible again. In the SolusVM panel needed to use the 'Reconfigure Networking' button. I got an error message ('An error occured: unklnown error') from the panel, but after that the VPS got accessible again.
Sadly the connectivity issues (routing problem?) still exist in many USA location For example: LAXA026, DALZ003, SEAZ010, etc. I'm able to reache most of the servers from my local (home) ISP, but the VM's can't connect to my other application servers (connection timed out, etc). For example: (ping.pe test):: https://i.imgur.com/gLsY4hx.png
I'm on LAXA015. I'm in the IP block with the issues. Based on earlier comments I tried connecting via VPN to a local server in Los Angeles. It works!
I can SSH in from my client computer and also access HTTPS (after bypassing my usual Cloudflare setup). Wanted to pass this on in case anyone can make use of it.
@FrankZ said:
I could be wrong but IIRC, reconfigure networking in SolusVM is designed to work with templates, not ISO installs. So it is a feature not a bug.
Comments
I'd be using Linux if I could get into one of my VPSs.
Seems like I did not miss anything while I was away. /s
My Amsterdam and Frankfurt VMs are working fine now with the new IPs.
Just a quick note for Alma 8 users....
I found each of my three EU VMs stuck at a waiting on rc.local job during boot after the old IPv4 were removed. While using VNC and booting into Alma 8 rescue mode from the boot menu, not rescue mode in SolusVM, I was able to start the VMs normally and change what needed to be changed regarding IPv4 configurations.
EDIT: Spoke too soon, both Frankfurt VMs are unreachable again.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
all the 47.87.xx.xx is gone forever it seems.
Oddly, Hetrix reports 47.87.196.xxx alternating between reachable and non-reachable, depending on which origin is attempting to hit it.
I guess it's a lingering BGP route?
P A N I C
all new IPs just went down, lmao/panic.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Only took 8 minutes from the time @FrankZ when online in Frankfurt.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
I just booted both mine up and they're good now.
Yeah, seems like something (or someone) shut down everything I had in Frankfurt/Amsterdam and I had to manually click BOOT in panel. Is this IP renumbering script going crazy?
Agent hasn't sent any data in the past: 16 minutes
Welp, there goes uptime
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Yep, they didn't even pull it correctly. Usually how it works is they would make de-announcement requests. They just unsigned the records, didn't pull it from ARIN. Which makes it easier to bring it back so whatever.
Frankfurt got rebooted. Amsterdam should still be in the same state as it was a few hours ago.
I guess this is new entry in Network status, but this needs timestamps in some known timezone, @VirMach please next time - put timestamps on updates.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Thank you. I rebooted and all is good again in Frankfurt.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Yea, it's weird... I have a 47.87.141.xxx server in LA running a database and my web server (different provider) has had no issues connecting to it all day. I can also SSH into all my 47.87.xxx.xxx US VPS's. But none of them can access my web server.
3vms to go now.
Vague on purpose as I'm not the one doing it.
I have someone helping out who's more interested in fixing everything than being graceful. It most likely means SolusVM is not being cooperative so he's instead using a hammer on the servers. We had to do it in a very non-SolusVM API way for it not to take half a day (which is funny since, you know... but that just means doing it the other way would've taken two.) Amsterdam didn't get rebooted, but VMs might be receiving additional work.
Frankfurt finally got rebooted, it got delayed until now.
I put most of my VMs on DHCP back during the first wave of re-IP's. I just click Reboot and * poof * I've got the new IP.
I'm actually so happy it happened this way. Sure it's annoying and confusing but at least we only got like under 1,000 tickets instead of probably 2K+
Only thing I miss about OpenVZ is the poof.
Does this mean I should be expecting an extended outage?
It means I don't have an answer for that question right now but hopefully should later tonight.
I have too many things bound to specific IPs for that to work for me. Sure I don't have to do it that way, but I sleep better at night.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
is AMS considered complete?
I bench YABS 24/7/365 unless it's a leap year.
I tried to connect to my web server from my local location, timed out.
Connected to an VPN in NY, tried again, it worked!
My VPS is at the node SEAZ007, maybe it's something related to route propagation, maybe the previous IPs are being reestablished?
The problem looks like solved in Germany location (FFME003) with the new IP address. My VM accessible again. In the SolusVM panel needed to use the 'Reconfigure Networking' button. I got an error message ('An error occured: unklnown error') from the panel, but after that the VPS got accessible again.
Sadly the connectivity issues (routing problem?) still exist in many USA location
For example: LAXA026, DALZ003, SEAZ010, etc. I'm able to reache most of the servers from my local (home) ISP, but the VM's can't connect to my other application servers (connection timed out, etc). For example: (ping.pe test):: https://i.imgur.com/gLsY4hx.png
AMS down again?
I bench YABS 24/7/365 unless it's a leap year.
I'm on LAXA015. I'm in the IP block with the issues. Based on earlier comments I tried connecting via VPN to a local server in Los Angeles. It works!
I can SSH in from my client computer and also access HTTPS (after bypassing my usual Cloudflare setup). Wanted to pass this on in case anyone can make use of it.
now i know +1 more thing that itches you.
AMS broke because
was auto reconfigured to
when it should have been
I bench YABS 24/7/365 unless it's a leap year.
was auto reconfigured to
when it should have been
I could be wrong but IIRC, reconfigure networking in SolusVM is designed to work with templates, not ISO installs. So it is a feature not a bug.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
yeah except i didnt do that, it was automated.
i configured my network manually
I bench YABS 24/7/365 unless it's a leap year.