@erk said:
Oddly, Hetrix reports 47.87.196.xxx alternating between reachable and non-reachable, depending on which origin is attempting to hit it.
I guess it's a lingering BGP route?
Yep, they didn't even pull it correctly. Usually how it works is they would make de-announcement requests. They just unsigned the records, didn't pull it from ARIN. Which makes it easier to bring it back so whatever.
@Jab said: all new IPs just went down, lmao/panic.
@FrankZ said: EDIT: Spoke too soon, both Frankfurt VMs are unreachable again.
Frankfurt got rebooted. Amsterdam should still be in the same state as it was a few hours ago.
Yea, it's weird... I have a 47.87.141.xxx server in LA running a database and my web server (different provider) has had no issues connecting to it all day. I can also SSH into all my 47.87.xxx.xxx US VPS's. But none of them can access my web server.
@Jab said: Yeah, seems like something (or someone) shut down everything I had in Frankfurt/Amsterdam and I had to manually click BOOT in panel. Is this IP renumbering script going crazy?
@Jab said: I guess this is new entry in Network status, but this needs timestamps in some known timezone, @VirMach please next time - put timestamps on updates.
Vague on purpose as I'm not the one doing it.
I have someone helping out who's more interested in fixing everything than being graceful. It most likely means SolusVM is not being cooperative so he's instead using a hammer on the servers. We had to do it in a very non-SolusVM API way for it not to take half a day (which is funny since, you know... but that just means doing it the other way would've taken two.) Amsterdam didn't get rebooted, but VMs might be receiving additional work.
Frankfurt finally got rebooted, it got delayed until now.
Just a quick note for Alma 8 users....
I found each of my three EU VMs stuck at a waiting on rc.local job during boot after the old IPv4 were removed. While using VNC and booting into Alma 8 rescue mode from the boot menu, not rescue mode in SolusVM, I was able to start the VMs normally and change what needed to be changed regarding IPv4 configurations.
I put most of my VMs on DHCP back during the first wave of re-IP's. I just click Reboot and * poof * I've got the new IP.
@soulchief said: Yea, it's weird... I have a 47.87.141.xxx server in LA running a database and my web server (different provider) has had no issues connecting to it all day. I can also SSH into all my 47.87.xxx.xxx US VPS's. But none of them can access my web server.
I'm actually so happy it happened this way. Sure it's annoying and confusing but at least we only got like under 1,000 tickets instead of probably 2K+
@rockinmusicgv said:
NYCB009 has been down for about a day and seems to be on a different IP block. Is there any indication when that node might be restored?
Yeah, it didn't come back long enough for us to be able to complete the planned emergency migration. It's difficult to get back online too, we've been working on that for a long time. Still speaking with RAID controller manufacturer but now it seems like there's a whole set of other issues.
I don't have any more information at the time.
Does this mean I should be expecting an extended outage?
I tried to connect to my web server from my local location, timed out.
Connected to an VPN in NY, tried again, it worked!
My VPS is at the node SEAZ007, maybe it's something related to route propagation, maybe the previous IPs are being reestablished?
The problem looks like solved in Germany location (FFME003) with the new IP address. My VM accessible again. In the SolusVM panel needed to use the 'Reconfigure Networking' button. I got an error message ('An error occured: unklnown error') from the panel, but after that the VPS got accessible again.
Sadly the connectivity issues (routing problem?) still exist in many USA location For example: LAXA026, DALZ003, SEAZ010, etc. I'm able to reache most of the servers from my local (home) ISP, but the VM's can't connect to my other application servers (connection timed out, etc). For example: (ping.pe test):: https://i.imgur.com/gLsY4hx.png
I'm on LAXA015. I'm in the IP block with the issues. Based on earlier comments I tried connecting via VPN to a local server in Los Angeles. It works!
I can SSH in from my client computer and also access HTTPS (after bypassing my usual Cloudflare setup). Wanted to pass this on in case anyone can make use of it.
@FrankZ said:
I could be wrong but IIRC, reconfigure networking in SolusVM is designed to work with templates, not ISO installs. So it is a feature not a bug.
@FrankZ said:
I could be wrong but IIRC, reconfigure networking in SolusVM is designed to work with templates, not ISO installs. So it is a feature not a bug.
yeah except i didnt do that, it was automated.
i configured my network manually
This was pinging/not blocked off from ping, right? Looking into our reconfigure script that runs in the background.
@FrankZ said:
I could be wrong but IIRC, reconfigure networking in SolusVM is designed to work with templates, not ISO installs. So it is a feature not a bug.
yeah except i didnt do that, it was automated.
i configured my network manually
This was pinging/not blocked off from ping, right? Looking into our reconfigure script that runs in the background.
im not sure what the question is, but i had configured manually the new IP correctly and VM was online.
suddenly it was reconfigured to eth0.
i manually reconfigured back to ens3 and works again now.
like what @FrankZ mentioned im using ISO install and not from template, so it may affect those on ISO.
@cybertech said: yes in my case (manual config) it is now pingable , tried from SG/TYO/LAX
Figured out the issue. The script doesn't do a heavy level of testing (e.g, running an entire ping.pe) and since it's a newly announced block routing isn't perfect and it could falsely mark one as not pinging if there's a little bit of loss.
I'm filtering out the list first using mass ping tool and then only inputting that into the script instead. Sorry about that.
Going to just wait it out and check for myself before getting everyone's hopes up on the network status page, but I don't mind getting everyone's hopes up here, unofficially.
All of it, except Frankfurt and Amsterdam which we moved off this morning/afternoon. It'll still take some time of course, it'll come in similarly to how it went out, not all suddenly.
@VirMach said:
All of it, except Frankfurt and Amsterdam which we moved off this morning/afternoon. It'll still take some time of course, it'll come in similarly to how it went out, not all suddenly.
huh?i think tyoc026 has no impact,but actually my network got edited too,turns into eth0, and waste my 1hour of sleep time to fix this.
how I fix this? waste 55mins for fixing nothing and 5mins later it recovered itself
@VirMach copy-paste from LET, I am to stupid to understand it, but maybe important
@jtk said: I saw reports of intermittent reachability. I suspect, a problem now is that QuadraNet is announcing a /20 in this prefix, but there are only ROAs with Alibaba's origin ASes published in the RPKI for the covered address space, which means many nets will be invalidating and rejecting the QuadraNet announcement. Let's see how long it takes people in the LES thread to figure this out. :-)
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
@Jab said: @VirMach copy-paste from LET, I am to stupid to understand it, but maybe important
@jtk said: I saw reports of intermittent reachability. I suspect, a problem now is that QuadraNet is announcing a /20 in this prefix, but there are only ROAs with Alibaba's origin ASes published in the RPKI for the covered address space, which means many nets will be invalidating and rejecting the QuadraNet announcement. Let's see how long it takes people in the LES thread to figure this out. :-)
so, if I am reading the service status page correctly, the payment issue or what ever with Alibaba has been resolved and the subnet is getting reassigned and it will just take time? if so, I have a lot of that and thank you for the updates
Okay so good news is that they're aware, but looks like it's going to take a little longer than the initial "we'll be done in ten minutes." They're keeping me posted.
@dgc1980 said:
so, if I am reading the service status page correctly, the payment issue or what ever with Alibaba has been resolved and the subnet is getting reassigned and it will just take time? if so, I have a lot of that and thank you for the updates
Comments
Thank you. I rebooted and all is good again in Frankfurt.
Yea, it's weird... I have a 47.87.141.xxx server in LA running a database and my web server (different provider) has had no issues connecting to it all day. I can also SSH into all my 47.87.xxx.xxx US VPS's. But none of them can access my web server.
3vms to go now.
Vague on purpose as I'm not the one doing it.
I have someone helping out who's more interested in fixing everything than being graceful. It most likely means SolusVM is not being cooperative so he's instead using a hammer on the servers. We had to do it in a very non-SolusVM API way for it not to take half a day (which is funny since, you know... but that just means doing it the other way would've taken two.) Amsterdam didn't get rebooted, but VMs might be receiving additional work.
Frankfurt finally got rebooted, it got delayed until now.
I put most of my VMs on DHCP back during the first wave of re-IP's. I just click Reboot and * poof * I've got the new IP.
I'm actually so happy it happened this way. Sure it's annoying and confusing but at least we only got like under 1,000 tickets instead of probably 2K+
Only thing I miss about OpenVZ is the poof.
Does this mean I should be expecting an extended outage?
It means I don't have an answer for that question right now but hopefully should later tonight.
I have too many things bound to specific IPs for that to work for me. Sure I don't have to do it that way, but I sleep better at night.
is AMS considered complete?
I bench YABS 24/7/365 unless it's a leap year.
I tried to connect to my web server from my local location, timed out.
Connected to an VPN in NY, tried again, it worked!
My VPS is at the node SEAZ007, maybe it's something related to route propagation, maybe the previous IPs are being reestablished?
The problem looks like solved in Germany location (FFME003) with the new IP address. My VM accessible again. In the SolusVM panel needed to use the 'Reconfigure Networking' button. I got an error message ('An error occured: unklnown error') from the panel, but after that the VPS got accessible again.
Sadly the connectivity issues (routing problem?) still exist in many USA location
For example: LAXA026, DALZ003, SEAZ010, etc. I'm able to reache most of the servers from my local (home) ISP, but the VM's can't connect to my other application servers (connection timed out, etc). For example: (ping.pe test):: https://i.imgur.com/gLsY4hx.png
AMS down again?
I bench YABS 24/7/365 unless it's a leap year.
I'm on LAXA015. I'm in the IP block with the issues. Based on earlier comments I tried connecting via VPN to a local server in Los Angeles. It works!
I can SSH in from my client computer and also access HTTPS (after bypassing my usual Cloudflare setup). Wanted to pass this on in case anyone can make use of it.
now i know +1 more thing that itches you.
AMS broke because
was auto reconfigured to
when it should have been
I bench YABS 24/7/365 unless it's a leap year.
was auto reconfigured to
when it should have been
I could be wrong but IIRC, reconfigure networking in SolusVM is designed to work with templates, not ISO installs. So it is a feature not a bug.
yeah except i didnt do that, it was automated.
i configured my network manually
I bench YABS 24/7/365 unless it's a leap year.
This was pinging/not blocked off from ping, right? Looking into our reconfigure script that runs in the background.
im not sure what the question is, but i had configured manually the new IP correctly and VM was online.
suddenly it was reconfigured to eth0.
i manually reconfigured back to ens3 and works again now.
like what @FrankZ mentioned im using ISO install and not from template, so it may affect those on ISO.
I bench YABS 24/7/365 unless it's a leap year.
Node Name: TPAZ003
https://imgur.com/a/7qdj9Da
@cybertech would we be able to ping the public IPv4 address? As in no firewall preventing that?
yes in my case (manual config) it is now pingable , tried from SG/TYO/LAX
I bench YABS 24/7/365 unless it's a leap year.
Change eth0 to ens3
Restart network solve the problem for Frankfurt location.
https://microlxc.net/
Figured out the issue. The script doesn't do a heavy level of testing (e.g, running an entire ping.pe) and since it's a newly announced block routing isn't perfect and it could falsely mark one as not pinging if there's a little bit of loss.
I'm filtering out the list first using mass ping tool and then only inputting that into the script instead. Sorry about that.
Received an update from them, verbatim:
Going to just wait it out and check for myself before getting everyone's hopes up on the network status page, but I don't mind getting everyone's hopes up here, unofficially.
Nice, I see it back on RADB for a few so far.
47.87 prefix, or few subnets of it?
All of it, except Frankfurt and Amsterdam which we moved off this morning/afternoon. It'll still take some time of course, it'll come in similarly to how it went out, not all suddenly.
@VirMach
https://billing.virmach.com/serverstatus.php
It does not mention TPAZ003 and IP 47.87.206.*, the situation is the VM is not reachable.
Lets hope everything goes smoothly
That's within 47.87.128.0/17
TYOC007 just pop'd back online. Thanks VirMach.
Any news for 47.87.136.xxx and 47.87.153.xxx ?
https://microlxc.net/
--
47.87.128.0/17 = FROM 47.87.128.0 TO 47.87.255.255
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
huh?i think tyoc026 has no impact,but actually my network got edited too,turns into eth0, and waste my 1hour of sleep time to fix this.
how I fix this? waste 55mins for fixing nothing and 5mins later it recovered itself
@VirMach copy-paste from LET, I am to stupid to understand it, but maybe important
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Sounds like they're describing the initial issue?
(edit) Oh
so, if I am reading the service status page correctly, the payment issue or what ever with Alibaba has been resolved and the subnet is getting reassigned and it will just take time? if so, I have a lot of that and thank you for the updates
Okay so good news is that they're aware, but looks like it's going to take a little longer than the initial "we'll be done in ten minutes." They're keeping me posted.
Yep, basically.
I meant. didn't see any news for .136 and .153 but from .225 which is on the same range.
https://microlxc.net/
Anyone's machine in NY up? Mine is still down.
Powerful AMD Ryzen VPS (aff)
NY has just come back up for me.
All my services are just popping back online
Mine is also up now.
Powerful AMD Ryzen VPS (aff)
Seattle and Chicago also up!
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
Indeed, all green here now. Yay!