Hi @VirMach ,
My server IP is 83.147.242.217 on node SJCZ012. Could you please check what issues it has and tell me the approximate repair time?
I haven't used this server for a very long time, and I'd like to start using it again now. By the way, this server was originally hosted on a Japan node before being migrated to the current one.
I don't need the data inside the server anymore. Please just help me power it on. Thank you!
@ahnlak said:
IP address changes seem to have finally killed my until-now-reliable VPS in NY.
Honestly not sure if I can be bothered in trying to resolve it or just accept that it's time has come /sigh
Same thing on my servers. I applied new IP on two of my servers and both are now unreachable from outside the network.
@ahnlak said:
IP address changes seem to have finally killed my until-now-reliable VPS in NY.
Honestly not sure if I can be bothered in trying to resolve it or just accept that it's time has come /sigh
Which server?
Mine are on NYCB038 and NYCB045.
When it doesn't work, if anyone sees my message here, please specify between you cannot ping it and if you checked it globally. If you cannot reach it, ticket with MTR (both directions preferred or one way to a public test server we can test back from) would be best and provide ID here.
It's pretty much confirmed from my investigations that it's an upstream issue. Seems like one of the edge routers is doing something wrong, so if that path gets handled by that router, that's where it runs into problems. It's been reported to them.
The reason I need to know if you checked globally (ping.pe) or not is so I can differentiate with it being this issue or potentially another issue that's on OUR end (as in, I did something stupid personally.)
I'm on NYCB045 too - totally unreachable from all sources on ping.pe.
mtr from the outside reaches as far as 45.59.169.xxx (something in royalehosting)
Annoyingly don't have mtr installed on the VPS (and I can't install it because... no network) but a ping to anywhere just gives "Destination is unreachable".
@ahnlak said: I'm on NYCB045 too - totally unreachable from all sources on ping.pe.
mtr from the outside reaches as far as 45.59.169.xxx (something in royalehosting)
Annoyingly don't have mtr installed on the VPS (and I can't install it because... no network) but a ping to anywhere just gives "Destination is unreachable".
If it is 100% unreachable from everywhere, that's a different issue.
On NYCB045, I see some virtual servers that are definitely receiving connectivity (with the routing issue) but I also see at least one that got an early change and it's not pinging.
The one that I am checking, requires a manual reconfiguration (or at least clicking the reconfigure button, but it could be that it actually cannot reconfigure the OS.)
@ahnlak said: I'm on NYCB045 too - totally unreachable from all sources on ping.pe.
mtr from the outside reaches as far as 45.59.169.xxx (something in royalehosting)
Annoyingly don't have mtr installed on the VPS (and I can't install it because... no network) but a ping to anywhere just gives "Destination is unreachable".
If it is 100% unreachable from everywhere, that's a different issue.
On NYCB045, I see some virtual servers that are definitely receiving connectivity (with the routing issue) but I also see at least one that got an early change and it's not pinging.
The one that I am checking, requires a manual reconfiguration (or at least clicking the reconfigure button, but it could be that it actually cannot reconfigure the OS.)
That's probably not mine then - I clicked the reconfigure, and the network settings look ... reasonable?
@VirMach said: The one that I am checking, requires a manual reconfiguration (or at least clicking the reconfigure button, but it could be that it actually cannot reconfigure the OS.)
Tried setting the new IP locally, dead. Reverted back to the original IP and things are working.
Setting the machine to the new IP + reconfigure, dead.
Reverting back to the the original IP and reconfigure networking works.
Speaking for NYCB015, ping.pe has some routes fine and others 100% loss. The interesting part to me is that I appear to be able to get to one server but not the other on the same subnet in Chicago at both Hosthatch and TNA Hosting.
But $400 per year for 8TB nvme storage... that's quite a good deal...
It's been a long time and I was trying to sleep 😅 (BTW I saw a light at night a bit blurry, I thought it was the moon, but it was not supposed to be on that side of the sky, and right now I'm not sure if it was real or a dream).
@somik said:
But $400 per year for 8TB nvme storage... that's quite a good deal...
I wait for the $7/y/TB.
Nah, ok, will buy 2TB at $14/y/TB.
Spinning rust? oh yeaahhhh just put raidz-360 or something like that.
What are you taking about man? I got 2TB of spinning rust dedi from CC for $100/year and I am happy with it. . . well, I was until I saw @imok got 8TB of nvme for $400/year
And 4x3TB HDD.
And a /28 (it's pending but I trust @virmach will deliver any time in the future).
@imok said: @virmach about the dead 4TB nvme in my dedi in OKH, does it have a replacement? Just asking to see if I should open a ticket or not.
They exist.
I don't know if I made my comment earlier or not but if you want to speed up the process can you do some of the general confirmations for me? Like does it show up in BIOS, what does OS display, what kernel parameters set?
Nice. I'll see what I can get this weekend without burning the server.
Can't reach VPS after IP change and reboot (I had gotten an email saying there would be a node migration and new IP address). Did all the stuff with "reconfigure network", can't reach by VNC, little popup in the corner of the client area says "Powered off Node NYCB046" but it's unclear whether that means the node is powered off or that the VM running on the node is (virtually) powered off. Any idea if this node is down? If yes, no big deal, it's a super cheap VPS so I can just wait til the node is back online. If the node is up then I might need help troubleshooting the VM.
@skorous said:
Speaking for NYCB015, ping.pe has some routes fine and others 100% loss. The interesting part to me is that I appear to be able to get to one server but not the other on the same subnet in Chicago at both Hosthatch and TNA Hosting.
Yeah, it's strange.
I'm still investigating on our end but I've spent a lot of time on it without anything fruitful coming from my investigation. I'm just trying to make sure it's nothing that could theoretically be on our end, I still don't believe it is since all we're doing is sending them over to our provider's routers. My guess is that it's potentially not a problem on their end either but could be an issue at Arelion.
I'm sure once it's figured out it'll seem like it was obvious and I encourage anyone to chime in, I just have no idea what could be causing the situation you're describing and that's probably what's making it so difficult to diagnose, it's difficult to find a pattern when a situation such as "it reaches one server but not the other even though they're on the same subnet."
@dgc1980 said:
mine on NYCB044 also in unreachable from anywhere, it is not even able to ping the gateway .1 but I can ping the gateway from outside.
That sounds extra weird, I'll look into it.
@willie said:
Can't reach VPS after IP change and reboot (I had gotten an email saying there would be a node migration and new IP address). Did all the stuff with "reconfigure network", can't reach by VNC, little popup in the corner of the client area says "Powered off Node NYCB046" but it's unclear whether that means the node is powered off or that the VM running on the node is (virtually) powered off. Any idea if this node is down? If yes, no big deal, it's a super cheap VPS so I can just wait til the node is back online. If the node is up then I might need help troubleshooting the VM.
It's not down. I haven't yet changed the main nodes to the new blocks as I did not want to create additional issues. If it says powered off, and you cannot use VNC, it sounds like an issue your VPS is facing after it attempted to reboot during the reconfigure, as in a SVM bug. That's just a guess.
Message me your IP and I can confirm.
We have a system to automatically catch these but with the other overlapping issues going on, it's not really "built" for this environment and therefore is pretty useless right now until it is reworked. This is because SVM doesn't provide API for these particular issues so we have to "guess" that it's happening based on other data. I haven't thought of a good way of accounting for it when there's also routing issues.
I've been able to confirm something, after going through the data.
Let's say there are 20 different test points, and let's say that each block can reach 10 test points, but not the other 10 test points. We have 10x new /24 subnets (we actually have 11x but to keep it simple we're saying 10.)
The pattern is that 5x of the new /24 subnets (let's call them "Group A") can reach 10 test points (let's call them "Reachable A"), and not the other 10 test points ("Unreachable A"). Then, the other 5x /24 subnets ("Group B") can reach 10 test points ("Reachable B") and not the other 10 test points ("Unreachable B"). The discovery is that Unreachable A = Reachable B and Unreachable B = Reachable A.
And there appear to be two Arelion interconnects.
"Group A" and "Group B" have no relation to any internal grouping on our end. They can be on any VLAN, etc.
This was my original "guess" on what could be happening, but I've finally pretty much confirmed it as best as we can on our end. It's similar to what happened in San Jose about two years ago where there were two edge routers and they had different configurations. I've already reported this as a possibility so at this point I'm going to just wait for this to be fixed upstream. I didn't want to "box" myself in so I started looking into all the other possibilities, which made it difficult to figure out since there were two different distinct clusters with different patterns that might make it appear "random" if you're not careful.
Just a fun fact though, this means if you have two services in NYC on different nodes, it's theoretically statistically probable that you could be in a situation where you do have full internet access if you just combine them together.
@skorous said:
Speaking for NYCB015, ping.pe has some routes fine and others 100% loss. The interesting part to me is that I appear to be able to get to one server but not the other on the same subnet in Chicago at both Hosthatch and TNA Hosting.
Yeah, it's strange.
I'm still investigating on our end but I've spent a lot of time on it without anything fruitful coming from my investigation. I'm just trying to make sure it's nothing that could theoretically be on our end, I still don't believe it is since all we're doing is sending them over to our provider's routers. My guess is that it's potentially not a problem on their end either but could be an issue at Arelion.
I'm sure once it's figured out it'll seem like it was obvious and I encourage anyone to chime in, I just have no idea what could be causing the situation you're describing and that's probably what's making it so difficult to diagnose, it's difficult to find a pattern when a situation such as "it reaches one server but not the other even though they're on the same subnet."
@dgc1980 said:
mine on NYCB044 also in unreachable from anywhere, it is not even able to ping the gateway .1 but I can ping the gateway from outside.
That sounds extra weird, I'll look into it.
FWIW, that's my situation too - can mtr the gateway fine from the outside but no further, and the VPS itself considers the gateway "Destination Host Unreachable"
We've had some problems with their routers hanging on to stale routes well after they are no longer announced. If you are seeing something weird with Arelion, it could be related to this issue.
Both of mentioned VM / node unreachable, offline for many-many months.
There is any ETA for solution?
Edit:
My VPS from NYCB046 node (I've got new IP address for this VPS) also unreachable.
It worked fine before the IP change. (Networking config has been reconfigured, and it probably OK). IP range: 185.227.252.XXX Screenshot:https://iili.io/C3YNrcF.png
I don't know where my SanJose vm is. It went L.A, and then Japan, and now the i.p showing is 1.0.3.xxx, which is in China.
It might end up in North Korea.
@tenpera said:
I don't know where my SanJose vm is. It went L.A, and thæn Japan, and now the i.p showing is 1.0.3.xxx, which is in China.
It might end up in North Korea.
Your VM went to more countries thæn I have
I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.
@taizi said:
Your new IP is still being assigned. Please try again in a few minutes.
i was thought i could replace my IP on tyoc037 before maintenance🤦
Looks like things are stuck "Migration queued" - 2 VMs both with multiple IPs on AMSD018.
Duplicate emails received (very short notice) about IP change but only gave a single IP.
VMs are still running on 'old' ips.
Comments
@VirMach have same issue with mine - pinging on few routes, checked on ping.pe
Get some hosting at https://drserver.net .
Hi @VirMach ,
My server IP is 83.147.242.217 on node SJCZ012. Could you please check what issues it has and tell me the approximate repair time?
I haven't used this server for a very long time, and I'd like to start using it again now. By the way, this server was originally hosted on a Japan node before being migrated to the current one.
I don't need the data inside the server anymore. Please just help me power it on. Thank you!
I'm on NYCB045 too - totally unreachable from all sources on ping.pe.
mtr from the outside reaches as far as 45.59.169.xxx (something in royalehosting)
Annoyingly don't have mtr installed on the VPS (and I can't install it because... no network) but a ping to anywhere just gives "Destination is unreachable".
If it is 100% unreachable from everywhere, that's a different issue.
On NYCB045, I see some virtual servers that are definitely receiving connectivity (with the routing issue) but I also see at least one that got an early change and it's not pinging.
The one that I am checking, requires a manual reconfiguration (or at least clicking the reconfigure button, but it could be that it actually cannot reconfigure the OS.)
That's probably not mine then - I clicked the reconfigure, and the network settings look ... reasonable?
Ticket #425233 if it helps
mine on NYCB044 also in unreachable from anywhere, it is not even able to ping the gateway .1 but I can ping the gateway from outside.
If you can't even ping your own gateway are you sure the reconfigure button used the proper interface name?
it is configuring from DHCP no issues, even manually configuring the IP same issue.
even booting systemrescuecd configures from DHCP without any access to the outside world
Tried setting the new IP locally, dead. Reverted back to the original IP and things are working.
Setting the machine to the new IP + reconfigure, dead.
Reverting back to the the original IP and reconfigure networking works.
Speaking for NYCB015, ping.pe has some routes fine and others 100% loss. The interesting part to me is that I appear to be able to get to one server but not the other on the same subnet in Chicago at both Hosthatch and TNA Hosting.
As an aside, the network status page is really nice.
"Aside from that Mrs. Lincoln, how was the show?"
"It's a hard life- to be a stick insect." - Karl Pilkington
LOL
It's been a long time and I was trying to sleep 😅 (BTW I saw a light at night a bit blurry, I thought it was the moon, but it was not supposed to be on that side of the sky, and right now I'm not sure if it was real or a dream).
I will check the actual price.
And 4x3TB HDD.
And a /28 (it's pending but I trust @virmach will deliver any time in the future).
Nice. I'll see what I can get this weekend without burning the server.
Can't reach VPS after IP change and reboot (I had gotten an email saying there would be a node migration and new IP address). Did all the stuff with "reconfigure network", can't reach by VNC, little popup in the corner of the client area says "Powered off Node NYCB046" but it's unclear whether that means the node is powered off or that the VM running on the node is (virtually) powered off. Any idea if this node is down? If yes, no big deal, it's a super cheap VPS so I can just wait til the node is back online. If the node is up then I might need help troubleshooting the VM.
Yeah, it's strange.
I'm still investigating on our end but I've spent a lot of time on it without anything fruitful coming from my investigation. I'm just trying to make sure it's nothing that could theoretically be on our end, I still don't believe it is since all we're doing is sending them over to our provider's routers. My guess is that it's potentially not a problem on their end either but could be an issue at Arelion.
I'm sure once it's figured out it'll seem like it was obvious and I encourage anyone to chime in, I just have no idea what could be causing the situation you're describing and that's probably what's making it so difficult to diagnose, it's difficult to find a pattern when a situation such as "it reaches one server but not the other even though they're on the same subnet."
That sounds extra weird, I'll look into it.
It's not down. I haven't yet changed the main nodes to the new blocks as I did not want to create additional issues. If it says powered off, and you cannot use VNC, it sounds like an issue your VPS is facing after it attempted to reboot during the reconfigure, as in a SVM bug. That's just a guess.
Message me your IP and I can confirm.
We have a system to automatically catch these but with the other overlapping issues going on, it's not really "built" for this environment and therefore is pretty useless right now until it is reworked. This is because SVM doesn't provide API for these particular issues so we have to "guess" that it's happening based on other data. I haven't thought of a good way of accounting for it when there's also routing issues.
I've been able to confirm something, after going through the data.
Let's say there are 20 different test points, and let's say that each block can reach 10 test points, but not the other 10 test points. We have 10x new /24 subnets (we actually have 11x but to keep it simple we're saying 10.)
The pattern is that 5x of the new /24 subnets (let's call them "Group A") can reach 10 test points (let's call them "Reachable A"), and not the other 10 test points ("Unreachable A"). Then, the other 5x /24 subnets ("Group B") can reach 10 test points ("Reachable B") and not the other 10 test points ("Unreachable B"). The discovery is that Unreachable A = Reachable B and Unreachable B = Reachable A.
And there appear to be two Arelion interconnects.
"Group A" and "Group B" have no relation to any internal grouping on our end. They can be on any VLAN, etc.
This was my original "guess" on what could be happening, but I've finally pretty much confirmed it as best as we can on our end. It's similar to what happened in San Jose about two years ago where there were two edge routers and they had different configurations. I've already reported this as a possibility so at this point I'm going to just wait for this to be fixed upstream. I didn't want to "box" myself in so I started looking into all the other possibilities, which made it difficult to figure out since there were two different distinct clusters with different patterns that might make it appear "random" if you're not careful.
Just a fun fact though, this means if you have two services in NYC on different nodes, it's theoretically statistically probable that you could be in a situation where you do have full internet access if you just combine them together.
Thanks, PM'd.
I get the alert for 2 of my NY VPS'es:
NYCB014
NYCB015
In the spirit of "Lets break just because something is working fine" I wait for resolution
blog | exploring visually |
@VirMach TYOC033 IPV6 is unavailable
FWIW, that's my situation too - can mtr the gateway fine from the outside but no further, and the VPS itself considers the gateway "Destination Host Unreachable"
OMG my AMS instance might be usable after a year?
We've had some problems with their routers hanging on to stale routes well after they are no longer announced. If you are seeing something weird with Arelion, it could be related to this issue.
QuickPacket - Dedicated Servers in Ashburn, Los Angeles, Chicago
Dear @VirMach
There is any update with the following nodes(?):
ATLZ033 (IP: 66.59.196.XY) (Ticket ID: #968802 - Submitted: 10/15/2025 (13:22)
Screenshot: https://freeimage.host/i/C3YlpaV
SJCZ006 (IP: 91.186.210.XY) (Has no ticket)
Screenshot: https://freeimage.host/i/C3Y0bxn
Both of mentioned VM / node unreachable, offline for many-many months.
There is any ETA for solution?
Edit:
My VPS from NYCB046 node (I've got new IP address for this VPS) also unreachable.
It worked fine before the IP change. (Networking config has been reconfigured, and it probably OK).
IP range: 185.227.252.XXX
Screenshot: https://iili.io/C3YNrcF.png
Thank You very much for the help, answer!
Titus
Your new IP is still being assigned. Please try again in a few minutes.
i was thought i could replace my IP on tyoc037 before maintenance🤦
I don't know where my SanJose vm is. It went L.A, and then Japan, and now the i.p showing is 1.0.3.xxx, which is in China.
It might end up in North Korea.
Your VM went to more countries thæn I have
I got like 4 e-mails about new IP's. My previous post bragging about me missing on all the fun is now invalid.
Thanks VirMach!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Bug fixed.
Looks like things are stuck "Migration queued" - 2 VMs both with multiple IPs on AMSD018.
Duplicate emails received (very short notice) about IP change but only gave a single IP.
VMs are still running on 'old' ips.
Mike
Get your FREE VPS if you develop Open Source software