@VirMach said: (Edit2) I just went through all of them, they're ALL the same as last time.
Just ask them where is the legacy legacy ticket system that you need to answer for those because seems like answering via legacy ticket system is not enough, not even talking about current ticket system.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Please someone read what I posted, this is just one instance of us answering them, let me know if it's unclear. I explained everything thoroughly, no? If you disagree then feel free to critique it and let me know how it should be done instead. Was it not clear? If it wasn't let me know and I'll pull up some other times we answered them so we can see if those are clear.
I find that when people keep asking me for the same thing over and over again it is normally not because I was previously unclear, but because I did not tell them what they wanted to hear.
Tagging you @yoursunny since one of their C-level people tried to insinuate that we don't handle abuse reports.
Since I don't have time to screenshot everything, this is the case for the first IP address I posted the screenshot to earlier, let's go step by step.
The IP: 47.87.220.105
We respond to this ticket.
We get an email from a random email from someone's Hivelocity email, not from their ticket system or support email. Keep in mind I don't have much time so it's possible they're slightly off chronological order.
We receive another copy.
We receive a confirmation it has been closed.
The same person that confirmed it as resolved a day ago makes a new request.
We receive another copy of his new message from another email.
Someone else chimes in.
Someone else chimes in referencing all the tickets they made about this including the one where they confirm it is resolved. We respond again.
Then all the other stuff happens you guys already know about.
(edit) Remember, this is all for one instance of abuse, that was handled at the beginning. It doesn't include the follow-up emails we get asking to rate them and such.
@VirMach said:
Tagging you @yoursunny since one of their C-level people tried to insinuate that we don't handle abuse reports.
Since I don't have time to screenshot everything, this is the case for the first IP address I posted the screenshot to earlier, let's go step by step.
The IP: 47.87.220.105
We respond to this ticket.
We get an email from a random email from someone's Hivelocity email, not from their ticket system or support email. Keep in mind I don't have much time so it's possible they're slightly off chronological order.
We receive another copy.
We receive a confirmation it has been closed.
The same person that confirmed it as resolved a day ago makes a new request.
We receive another copy of his new message from another email.
Someone else chimes in.
Someone else chimes in referencing all the tickets they made about this including the one where they confirm it is resolved. We respond again.
Then all the other stuff happens you guys already know about.
We're going to try something different for NYC changes. When we make tickets it spams people and in this case there's no advanced notice on the IP change really, we already sent out an email asking to check status page and provided updates there.
There will be a pop-up you can't get rid of when you load your service details page instead telling you what happened.
Tracing route to tailscale.com [76.223.15.28]
over a maximum of 30 hops:
1 2 ms 4 ms 5 ms 46.233.44.1
2 11 ms 20 ms 20 ms 172.16.0.0
3 <1 ms <1 ms <1 ms hosted-by.royalehosting.net [45.59.169.176]
4 <1 ms <1 ms <1 ms 223.165.7.148
5 <1 ms <1 ms <1 ms po1.nyc-eqxny2-bb1.globalsecurelayer.com [206.148.25.61]
6 <1 ms <1 ms <1 ms ny.gslnetworks.com [198.32.118.59]
7 * * * Request timed out.
8 * * * Request timed out.
9
@hike said:
NY routing not going well lot of ASN unreachable.
Tracing route to tailscale.com [76.223.15.28]
over a maximum of 30 hops:
1 2 ms 4 ms 5 ms 46.233.44.1
2 11 ms 20 ms 20 ms 172.16.0.0
3 <1 ms <1 ms <1 ms hosted-by.royalehosting.net [45.59.169.176]
4 <1 ms <1 ms <1 ms 223.165.7.148
5 <1 ms <1 ms <1 ms po1.nyc-eqxny2-bb1.globalsecurelayer.com [206.148.25.61]
6 <1 ms <1 ms <1 ms ny.gslnetworks.com [198.32.118.59]
7 * * * Request timed out.
8 * * * Request timed out.
9
It'll be like this until the prefixes are RPKI validated. Basically Dedipath is still announcing the blocks even though they're not supposed to, and because of this some carriers that filter invalid RPKI will drop you at the edge. https://isbgpsafeyet.com/ explains this a little better than I am. This is also why some of Dedipath's IPs were accessible even though they were taken back by the prefix owners.
@hike said:
NY routing not going well lot of ASN unreachable.
Tracing route to tailscale.com [76.223.15.28]
over a maximum of 30 hops:
1 2 ms 4 ms 5 ms 46.233.44.1
2 11 ms 20 ms 20 ms 172.16.0.0
3 <1 ms <1 ms <1 ms hosted-by.royalehosting.net [45.59.169.176]
4 <1 ms <1 ms <1 ms 223.165.7.148
5 <1 ms <1 ms <1 ms po1.nyc-eqxny2-bb1.globalsecurelayer.com [206.148.25.61]
6 <1 ms <1 ms <1 ms ny.gslnetworks.com [198.32.118.59]
7 * * * Request timed out.
8 * * * Request timed out.
9
It'll be like this until the prefixes are RPKI validated. Basically Dedipath is still announcing the blocks even though they're not supposed to, and because of this some carriers that filter invalid RPKI will drop you at the edge. https://isbgpsafeyet.com/ explains this a little better than I am. This is also why some of Dedipath's IPs were accessible even though they were taken back by the prefix owners.
@JoeMerit said:
What does that mean? I have ATL & DEN servers. Does that mean youve restored them to another DC? You said LAX for DEN, where would you move ATL?
I'm not spending a lot of time on this, but from one of the many systems I have that are unreachable, you're probably seeing this:
Notice the "RPKI State Invalid" part. You might get partial reachability behind those networks that don't enforce ROV (route origin validation), which is a relatively small portion of the entire Internet these days when you're a low-end provider. In a word, routing is FUBAR'd until the cooperating parties are in sync on who can legitimately announce what and populate the respective IRRs and RPKI repository with that info.
If you have your own address space, you're not at the mercy of third party address brokers, but presumably that is not an easy/cheap solution. But, it is something I increasing pay attention to these days or ask about.
@VirMach said:
Since I don't have time to screenshot everything, this is the case for the first IP address I posted the screenshot to earlier, let's go step by step.
This reminds me of most of my conversations with providers about IPv6 issues.
Better off fixing it manually, IME, due to network interface naming - especially if you use static IP, which you should on a server.
[Edit for clarity.]
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
14/35/36 have new IPs, some of those are "close" to OLD IPs - like I have one in 46.X.X.X so I discarded it some time ago as "not updated, still the same range", but it's not
also it's still hit & miss
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
My NYCB030 VPS is now up as I can see it via VNC and has same IP but can't SSH.
Restart via VNC but same issue. I can log into machine via VNC. And I can see it is assigned same IP when I ran command ip a (and same is shown in billing panel).
@lesuser said:
My NYCB030 VPS is now up as I can see it via VNC and has same IP but can't SSH.
Restart via VNC but same issue. I can log into machine via VNC. And I can see it is assigned same IP when I ran command ip a (and same is shown in billing panel).
Have you read the part of the thread where people are talking about the routing issues?
My NYCB015 VM has had a new IP assigned, I did not need to reconfigure networking.
I can get to it via ssh through a jump box I have at VirMach Amsterdam without using VNC.
Can ping 8.8.8.8 and 9.9.9.9.
Most other places I still need to wait for route updates.
Huge ticket backlog did eventually build up. All custom created tickets will likely be auto closed, any created using the automatic creator where it assigns an appropriate title will be looked through still. Haven't decided yet.
Keep in mind anyone who opened a ticket about an existing network issue will be flagged accordingly. It just tells us you ignore network status page and helps us handle support queue in the future by skipping your outage reports until all others are handled first, doesn't negatively affect your account in any other way, but if you're seeing this and you opened any ticket about something in anyway related to a node described in network status page, last call to close your ticket.
This is inaccurate, just making an important distinction: not every VM on every server will be done, those may still need to move around to get their new IP address. This is a very small number of services and we're aware ad working on it actively. We just need the last few I mentioned above to complete to proceed with that, as for example, NYCB043 will be receiving a good number of them.
(edit) I believe NYCB020 was one of the ones we specifically listed here as needing to be moved, in which case it means about 20% of them need this change still.
As for our plans outside of NYC, I'll provide it in greater detail below, I'm still working on that update for you guys, but some of it needs to be changed as we wanted to use Hivelocity which we most definitely will not be doing after what happened yesterday. So we're trying to make up for the units we can no longer use there with them, because I sure as hell am not sending more servers to be trapped there when they're behaving the way they are, they definitely have proven themselves to be the type of company that would potentially hold our hardware hostage while simultaneously ending service. Of course I'm not saying that's definitely the case, just based on my interpretation of everything that's happened so far over 13 abuse reports across like 3 to 4 months.
(edit) There is still a small chance we do, but only after I speak with our account manager and make it clear we're temporarily placing servers here and will need to take them out after and get that in writing.
This is inaccurate, just making an important distinction: not every VM on every server will be done, those may still need to move around to get their new IP address. This is a very small number of services and we're aware ad working on it actively. We just need the last few I mentioned above to complete to proceed with that, as for example, NYCB043 will be receiving a good number of them.
(edit) I believe NYCB020 was one of the ones we specifically listed here as needing to be moved, in which case it means about 20% of them need this change still.
Comments
Just ask them where is the legacy legacy ticket system that you need to answer for those because seems like answering via legacy ticket system is not enough, not even talking about current ticket system.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Please someone read what I posted, this is just one instance of us answering them, let me know if it's unclear. I explained everything thoroughly, no? If you disagree then feel free to critique it and let me know how it should be done instead. Was it not clear? If it wasn't let me know and I'll pull up some other times we answered them so we can see if those are clear.
I find that when people keep asking me for the same thing over and over again it is normally not because I was previously unclear, but because I did not tell them what they wanted to hear.
Tagging you @yoursunny since one of their C-level people tried to insinuate that we don't handle abuse reports.
Since I don't have time to screenshot everything, this is the case for the first IP address I posted the screenshot to earlier, let's go step by step.
The IP: 47.87.220.105
We respond to this ticket.
We get an email from a random email from someone's Hivelocity email, not from their ticket system or support email. Keep in mind I don't have much time so it's possible they're slightly off chronological order.
We receive another copy.
We receive a confirmation it has been closed.
The same person that confirmed it as resolved a day ago makes a new request.
We receive another copy of his new message from another email.
Someone else chimes in.
Someone else chimes in referencing all the tickets they made about this including the one where they confirm it is resolved. We respond again.
Then all the other stuff happens you guys already know about.
(edit) Remember, this is all for one instance of abuse, that was handled at the beginning. It doesn't include the follow-up emails we get asking to rate them and such.
I dun't knows nuffin and it is better that way.
The Yeti has left the building.
Bonus, I like this email I just sent out like 30 minutes ago.
I genuinely started highlighting them to see what new report was there to handle it and then this happened.
But... you should colour code group the same ticket numbers together!
How is he supposed to figure it out if everything is red. :-D
Oh update, I didn't handle this and I've been too busy to check communications.
ATLZ006, ATLZ005, DENZ001 and partial DENZ002 have been loaded in from backups.
Not sure red was a good color choice.
I'm thinking of things like:
Making them see red
Waving a red cloth in front of a bull
We're going to try something different for NYC changes. When we make tickets it spams people and in this case there's no advanced notice on the IP change really, we already sent out an email asking to check status page and provided updates there.
There will be a pop-up you can't get rid of when you load your service details page instead telling you what happened.
My VPS in San Jose is offline, I love the node and I'll wait quietly for it to come back online.
Some praying might help too.
I can imagine this gonna be perfect for mobile users - whole screen ruined.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Operation Timed Out After 90001 Milliseconds With 0 Bytes Received
What does that mean? I have ATL & DEN servers. Does that mean youve restored them to another DC? You said LAX for DEN, where would you move ATL?
NY routing not going well lot of ASN unreachable.
Tracing route to tailscale.com [76.223.15.28]
over a maximum of 30 hops:
1 2 ms 4 ms 5 ms 46.233.44.1
2 11 ms 20 ms 20 ms 172.16.0.0
3 <1 ms <1 ms <1 ms hosted-by.royalehosting.net [45.59.169.176]
4 <1 ms <1 ms <1 ms 223.165.7.148
5 <1 ms <1 ms <1 ms po1.nyc-eqxny2-bb1.globalsecurelayer.com [206.148.25.61]
6 <1 ms <1 ms <1 ms ny.gslnetworks.com [198.32.118.59]
7 * * * Request timed out.
8 * * * Request timed out.
9
It'll be like this until the prefixes are RPKI validated. Basically Dedipath is still announcing the blocks even though they're not supposed to, and because of this some carriers that filter invalid RPKI will drop you at the edge. https://isbgpsafeyet.com/ explains this a little better than I am. This is also why some of Dedipath's IPs were accessible even though they were taken back by the prefix owners.
tldr give it a little bit
Thanks, let see how fast they get validated.
VirMach'd fast!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
alright bro. hope data is safe
I'm not spending a lot of time on this, but from one of the many systems I have that are unreachable, you're probably seeing this:
Notice the "RPKI State Invalid" part. You might get partial reachability behind those networks that don't enforce ROV (route origin validation), which is a relatively small portion of the entire Internet these days when you're a low-end provider. In a word, routing is FUBAR'd until the cooperating parties are in sync on who can legitimately announce what and populate the respective IRRs and RPKI repository with that info.
If you have your own address space, you're not at the mercy of third party address brokers, but presumably that is not an easy/cheap solution. But, it is something I increasing pay attention to these days or ask about.
Dataplane.org's current server hosting provider list
5 more months and I'm VirTrash free ! inshallah!
↪️ Service Transfers
You can get rid of VirmAche faster.
We accept Karma donations for the last flan. 🍮 affbrr
This reminds me of most of my conversations with providers about IPv6 issues.
dnscry.pt - Public DNSCrypt resolvers hosted by LowEnd providers • Need a free NAT LXC? -> https://microlxc.net/
I like to feel the pain for a few months
NYCB022 can be acessed via VNC but i cant ping in or out, should i try the fix network button?
Better off fixing it manually, IME, due to network interface naming - especially if you use static IP, which you should on a server.
[Edit for clarity.]
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
didn't worked for me
LMFTFY
See above.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I Just updated the /etc/network/interfaces
with the new ip/gateway and now its working again
its not VPS its IP propagation, I suggest you wait
What nodes have new IPs?
NYCB028 have still old IPs.
14/35/36 have new IPs, some of those are "close" to OLD IPs - like I have one in 46.X.X.X so I discarded it some time ago as "not updated, still the same range", but it's not
also it's still hit & miss
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
LAXA018 has been down for a day without listed on network status page
All of Seattle nodes are down again. 23 hours passed,
I work at https://osd.vn. I work for https://thienphat.com.vn and https://luatsuanviet.com
My NYCB030 VPS is now up as I can see it via VNC and has same IP but can't SSH.
Restart via VNC but same issue. I can log into machine via VNC. And I can see it is assigned same IP when I ran command
ip a(and same is shown in billing panel).Powerful AMD Ryzen VPS (aff)
Have you read the part of the thread where people are talking about the routing issues?
New IPs, not his.
He reported that he (node NYCB030 ) have still the same (old) IP. The same as NYCB028 and most likely several others.
My NYCB015 VM has had a new IP assigned, I did not need to reconfigure networking.
I can get to it via ssh through a jump box I have at VirMach Amsterdam without using VNC.
Can ping 8.8.8.8 and 9.9.9.9.
Most other places I still need to wait for route updates.
Is it still IP from aliababa/Zenlayer subnet?
IPXO
Connectivity restored to remaining 3 NYC, IP changes will proceed if not already completed.
Oh also 3 others were waiting for the above 3 to be able to proceed in some ways so completing those too.
Will provide a more detailed plan for other locations loading in from backups after NYC is closed out.
Huge ticket backlog did eventually build up. All custom created tickets will likely be auto closed, any created using the automatic creator where it assigns an appropriate title will be looked through still. Haven't decided yet.
Keep in mind anyone who opened a ticket about an existing network issue will be flagged accordingly. It just tells us you ignore network status page and helps us handle support queue in the future by skipping your outage reports until all others are handled first, doesn't negatively affect your account in any other way, but if you're seeing this and you opened any ticket about something in anyway related to a node described in network status page, last call to close your ticket.
NYCB020 still on old IPs
This is inaccurate, just making an important distinction: not every VM on every server will be done, those may still need to move around to get their new IP address. This is a very small number of services and we're aware ad working on it actively. We just need the last few I mentioned above to complete to proceed with that, as for example, NYCB043 will be receiving a good number of them.
(edit) I believe NYCB020 was one of the ones we specifically listed here as needing to be moved, in which case it means about 20% of them need this change still.
As for our plans outside of NYC, I'll provide it in greater detail below, I'm still working on that update for you guys, but some of it needs to be changed as we wanted to use Hivelocity which we most definitely will not be doing after what happened yesterday. So we're trying to make up for the units we can no longer use there with them, because I sure as hell am not sending more servers to be trapped there when they're behaving the way they are, they definitely have proven themselves to be the type of company that would potentially hold our hardware hostage while simultaneously ending service. Of course I'm not saying that's definitely the case, just based on my interpretation of everything that's happened so far over 13 abuse reports across like 3 to 4 months.
(edit) There is still a small chance we do, but only after I speak with our account manager and make it clear we're temporarily placing servers here and will need to take them out after and get that in writing.
Its 100% accurate for me