@VirMach said: Okay so I've kind of ballpark divided everything out that we have available right now, it'll look something like this for anyone interested...
Okay.. but where the hell are the server pr0n pics?
Doesn't look too good right now. Need to freshen up a little, maybe put on a cute dress and then I'll try to remember. I'll let you decide if the dress is for the servers or myself.
@JoeMerit said: @VirMach how about a look at ATLZ005? high pings, high packet loss and speeds are like 10mbit/sec tops.
Let's quickly set the stage here. We sent 10 servers to Atlanta. Four of them broke in one way or another, two motherboards got lost, a server got lost, we probably had to make over a dozen hands requests to try to troubleshoot any of it which resulted in no resolution. I don't even think we could get a KVM attached in this location.
Now I'm not saying it's the DC, it could just be cursed. But the fact that ATLZ005 is alive and existent is a miracle.
Okay now with that out of the way, I can confirm the issue you've described. Let me see what magic I can perform but it might be the switch port.
@VirMach said: The request did get sent to the DC but it's not even related to the original transferring. The OS reload was completed, you asked about an account merge in a weird way, which I mean it was from the beginning so I tried explaining how it has to be requested from the account you want merged into that one, since you mentioned that account being merged into the one you're creating a ticket about. We can't merge anyone who asks for it by mentioning another account email or else anyone could request anyone's account to be merged into theirs. I was just saying it has to be requested from the account getting merged into another which is still true. You didn't mention that the account already had a ticket to get merged into this one.
no, I opened a ticket asking for reinstall first, then opened another ticket asking for service transfer status. then your script merge these together, just like these old tickets.
I think I will have merge PTSD, and let me only have one active ticket at the same time...
@VirMach said: Now I'm not saying it's the DC, it could just be cursed.
@FrankZ goes to the client panel to check his Atlanta VM and is delighted to see it is not a "Special" but a "Legacy KVM Plan SSD4G" with the migration button enabled. He clicks button to see what location options are available and when he sees this.
He knows @VirMach is right.
47.87.131. Los Angeles
47.87.150. Los Angeles
47.87.187. Seattle
All show cloudflare time error on rDNS popup in SolusVM. I expect VirMach is doing maintenance or making adjustments to the system.
A timeout occurred Error code 524
Visit cloudflare.com for more information.
2023-03-30 05:31:16 UTC
You
Browser Working
Dallas
Cloudflare Working
solusvm.virmach.com
Host Error
@VirMach said: Okay now with that out of the way, I can confirm the issue you've described. Let me see what magic I can perform but it might be the switch port.
Thanks, you worked magic and its back to being perfect.
I have been running three geo-located redundant web servers in the Asia region, Stromonic in Mumbai, WebHorizons in Singapore, and VirMach in Tokyo. The only one that is still currently working is VirMach Tokyo.
Also the "maintenance" for the rDNS seems to be complete and this is now working correctly and instantaneously for me.
This one lets you adjust some things and keep a history (until it crashes then all history is lost.) Also lets you save/share the results and looks nicer, as well as letting you set test duration and such. I think we'll potentially set up a cluster for this one so each location gets most our servers in that datacenter for testing later so it's less congested per port. Only issue with this one is it's more finnicky with SSL so results may vary, I haven't actually tried capping it out yet to see the bottleneck on SSL.
Test files have been removed as I don't want to deal with the abuse on that, I'll see about adding CLI test or iPerf, later.
I also wanted to try adding some type of multi-location traceroute tool, potentially with visual hops on a world map, but all the ones I've found so far are outdated and not functional. Feel free to recommend if anything comes to mind. that we can deploy. It'd be cool to use if customers can do it from their specific servers and get us results versus having to use MTR, more user friendly in finding problems.
Oh speaking of routing issues we had one with HE that's been resolved as well as the magic done on the node JoeMerit reported.
Miami is offline since an hour ago. Operation Timed Out After 90001 Milliseconds With 0 Bytes Received
I don't know which node because my records are stored in Seafile hosted on this node.
I can't peek into the backup server because the decryption key is stored in Seafile.
Something's wrong with my backup arrangement.
Update: found an unencrypted backup of my notes.
It's MIAZ011.
Priority Ticket #225283
MIAZ011: the host operating system may be corrupt. We're still attempting to bring the server back online but may need to proceed wit emergency migrations.
I expect you already know the answer to this question ....
Specials have low prices, but limited extra features.
All of my non limited special (aka regular) VMs are full featured, with all the buttons. So if you want these options they are available at a slightly higher price point.
@yoursunny said: MIAZ011: the host operating system may be corrupt. We're still attempting to bring the server back online but may need to proceed wit emergency migrations.
So, everyone is dead, all data is lost?
Having a hard time with this one, mainly limited on time since everything happened all at once again and ASRock IPMI being not great. Data corruption is on the OS drive, not the volume that holds customer's VM data, at least seems that way so far. I've had to work on other things for last few hours but hopefully can resume on this soon.
This server has way too many drives so we'll be able to get it back up in any case without hardware replacements. I don't know how the hell this happened but it has 4x2TB NVMe on a riser, 2x2TB NVMe on M.2 slots, and 3x 1TB SATA SSDs. I guess I had fun with this one.
(edit) Which is kind of funny now that I think about it because one of the servers in Miami has a lack of drives. I wonder if that's related. It actually may be... I think one or two of them came from HV so maybe the riser got placed into the wrong server. Anyway that's a problem for another day.
Small question @VirMach, with those 6 NVMe configurations, did you ever experience drives just randomly dropping out on X470/X570 boards? I've tested a few configurations with 4 x 500GB Gen4 NVMe on the riser and 2 on the motherboard on various motherboards and setups, but they just kept on dropping out for absolutely no reason whatsoever, at random, and then they would magically reappear after a reboot. I've heard 1 or 2 other providers/people with the exact same issue
Comments
10 to Tokyo?
standing by for deals
I bench YABS 24/7/365 unless it's a leap year.
rdns request kept spinning, and spinning...
Can you please add location and IP subnet ?
This is SolusVM correct ?
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Doesn't look too good right now. Need to freshen up a little, maybe put on a cute dress and then I'll try to remember. I'll let you decide if the dress is for the servers or myself.
Let's quickly set the stage here. We sent 10 servers to Atlanta. Four of them broke in one way or another, two motherboards got lost, a server got lost, we probably had to make over a dozen hands requests to try to troubleshoot any of it which resulted in no resolution. I don't even think we could get a KVM attached in this location.
Now I'm not saying it's the DC, it could just be cursed. But the fact that ATLZ005 is alive and existent is a miracle.
Okay now with that out of the way, I can confirm the issue you've described. Let me see what magic I can perform but it might be the switch port.
no, I opened a ticket asking for reinstall first, then opened another ticket asking for service transfer status. then your script merge these together, just like these old tickets.
I think I will have merge PTSD, and let me only have one active ticket at the same time...
@FrankZ goes to the client panel to check his Atlanta VM and is delighted to see it is not a "Special" but a "Legacy KVM Plan SSD4G" with the migration button enabled. He clicks button to see what location options are available and when he sees this.
He knows @VirMach is right.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
When I tried changing the rDNS:
rDNS requests for IP addresses in the subnet 47.87.0.0/16 can be self-managed via the SolusVM control panel.
I'm on "LAXA013" node.
Sign up at Dynadot [aff], spend $9.99 within 48 hours, and get $5.00 DynaDollars for domain registrations!
47.87.131. Los Angeles
47.87.150. Los Angeles
47.87.187. Seattle
All show cloudflare time error on rDNS popup in SolusVM. I expect VirMach is doing maintenance or making adjustments to the system.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
This is good news, but for now it is still not possible to use the SolusVM panel, which still prompts:
Wait a few months and check again for new messages.
from google translate
Thanks, you worked magic and its back to being perfect.
Ah yes "maintenance." Let's call it that and leave it there.
I'm re-importing the previously set rDNS. May take some time, so if you see this and it's not working just re-submit it.
It looks like the button placement is a bit wrong.
https://i.imgur.com/If6MWRg.png
I have been running three geo-located redundant web servers in the Asia region, Stromonic in Mumbai, WebHorizons in Singapore, and VirMach in Tokyo. The only one that is still currently working is VirMach Tokyo.
Also the "maintenance" for the rDNS seems to be complete and this is now working correctly and instantaneously for me.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
@VirMach any chance Tampa users get "Migration" button for Miami? :-)
All LG's have been updated. Slightly cleaner look, wrapping up last few locations now.
https://tyo.lg.virmach.com
Also trying out a new type of speed test script, turning that into it's own page.
https://virmach.com/speedtest/
This one lets you adjust some things and keep a history (until it crashes then all history is lost.) Also lets you save/share the results and looks nicer, as well as letting you set test duration and such. I think we'll potentially set up a cluster for this one so each location gets most our servers in that datacenter for testing later so it's less congested per port. Only issue with this one is it's more finnicky with SSL so results may vary, I haven't actually tried capping it out yet to see the bottleneck on SSL.
Test files have been removed as I don't want to deal with the abuse on that, I'll see about adding CLI test or iPerf, later.
I also wanted to try adding some type of multi-location traceroute tool, potentially with visual hops on a world map, but all the ones I've found so far are outdated and not functional. Feel free to recommend if anything comes to mind. that we can deploy. It'd be cool to use if customers can do it from their specific servers and get us results versus having to use MTR, more user friendly in finding problems.
Oh speaking of routing issues we had one with HE that's been resolved as well as the magic done on the node JoeMerit reported.
Yeah we'll add the function back for all HV services.
Thanks for reporting.
I figured something was wrong since people were making weird tickets on suspensions for BW. I'll have a look, added to the todo list.
Finally the route got fixed. I reported it half month ago.
service transfer when
seems like services are mostly stable!
I bench YABS 24/7/365 unless it's a leap year.
Weekly check whether Virmach ran away.
Why flash special can not be migrated!
Ontario Dildo Inspector
Miami is offline since an hour ago.
Operation Timed Out After 90001 Milliseconds With 0 Bytes Received
I don't know which node because my records are stored in Seafile hosted on this node.
I can't peek into the backup server because the decryption key is stored in Seafile.
Something's wrong with my backup arrangement.
Update: found an unencrypted backup of my notes.
It's MIAZ011.
Priority Ticket #225283
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
MIAZ011: the host operating system may be corrupt. We're still attempting to bring the server back online but may need to proceed wit emergency migrations.
So, everyone is dead, all data is lost?
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
I expect you already know the answer to this question ....
Specials have low prices, but limited extra features.
All of my non limited special (aka regular) VMs are full featured, with all the buttons. So if you want these options they are available at a slightly higher price point.
By the way how did your exam go ?
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Having a hard time with this one, mainly limited on time since everything happened all at once again and ASRock IPMI being not great. Data corruption is on the OS drive, not the volume that holds customer's VM data, at least seems that way so far. I've had to work on other things for last few hours but hopefully can resume on this soon.
This server has way too many drives so we'll be able to get it back up in any case without hardware replacements. I don't know how the hell this happened but it has 4x2TB NVMe on a riser, 2x2TB NVMe on M.2 slots, and 3x 1TB SATA SSDs. I guess I had fun with this one.
(edit) Which is kind of funny now that I think about it because one of the servers in Miami has a lack of drives. I wonder if that's related. It actually may be... I think one or two of them came from HV so maybe the riser got placed into the wrong server. Anyway that's a problem for another day.
@VirMach
Thank You!
Made my day happy with the rDNS option
The rDNS settings worked on SolusVM panel for one of my VM's IP from 47.87.226.XX range
Regarding MIAZ011 with a Ryzen 5950x, 128 GB RAM, 15TB NVMe/SSD there must be some mighty strange plans on that node.
Next flash sale ....
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Knowing VirBot, it'd probably end up being more like...
Small question @VirMach, with those 6 NVMe configurations, did you ever experience drives just randomly dropping out on X470/X570 boards? I've tested a few configurations with 4 x 500GB Gen4 NVMe on the riser and 2 on the motherboard on various motherboards and setups, but they just kept on dropping out for absolutely no reason whatsoever, at random, and then they would magically reappear after a reboot. I've heard 1 or 2 other providers/people with the exact same issue
I am a representative of Advin Servers
I spent all night restoring Seafile onto another server.
No data was lost in Seafile.
Now I hope the VM data is lost, so I can read drama threads.
Accepting submissions for IPv6 less than /64 Hall of Incompetence.