According to the status page DALZ007 is/was in the process of being migrated as of yesterday to NYC.
The VirMach comments that I linked to were interpreted by me as saying that Dallas is not going to be a VirMach location, at least in the near term, and it was unknown if that location would come back in the longer term.
According to the status page DALZ007 is/was in the process of being migrated as of yesterday to NYC.
The VirMach comments that I linked to were interpreted by me as saying that Dallas is not going to be a VirMach location, at least in the near term, and it was unknown if that location would come back in the longer term.
What am I missing ?
My bad I need to stop and go do something else. I actually thought he said Dallas was bad for him and could it be moved to Dallas.
I do not like myself at the moment. I need to go to bed and forget everything for about 15-16 hours. I think I may have blew the ram right out of my brain.
@AuroraZero said: My bad I need to stop and go do something else. I actually thought he said Dallas was bad for him and could it be moved to Dallas.
I do not like myself at the moment. I need to go to bed and forget everything for about 15-16 hours. I think I may have blew the ram right out of my brain.
Just remember that we like you, even when you might not like yourself.
I have no issue with what you said , and honestly thought I was the one missing something.
Of course getting some sleep is never a bad thing, given that you have been hitting it pretty hard for some time now.
(according to my stalker cam)
@reb0rn said: I have one VPS transfered to NYCB011X show OS but no boot, and looks like reinstall is no go (as I do not need data)
Does it show an O/S in solusVM, or just the billing panel ?
Yeah it's there also
If I try VNC it show: Authentication failed...
Interesting, I also get "Authentication failed", both before and after changing the password for VNC, for my broken migrated VM on NYCB45. This seems to be effecting more than just a couple of VMs so I would expect that the VirMach crew is aware and looking for a solution.
@AuroraZero said: My bad I need to stop and go do something else. I actually thought he said Dallas was bad for him and could it be moved to Dallas.
I do not like myself at the moment. I need to go to bed and forget everything for about 15-16 hours. I think I may have blew the ram right out of my brain.
Just remember that we like you, even when you might not like yourself.
I have no issue with what you said , and honestly thought I was the one missing something.
Of course getting some sleep is never a bad thing, given that you have been hitting it pretty hard for some time now.
(according to my stalker cam)
Yeah Virmach got me again. This is brutal anymore. Time to go with the big boys for security I suppose.
NYC connectivity should be up for the new blocks/migrations now. I'm just checking it further and will then update the network status page. Phoenix has been racked and should be ready, but BGP hasn't updated yet, I'm waiting to see if it slowly starts to show signs of life but I won't be able to check it any other way as I believe IPMI is still waiting.
San Jose... very long story. "Short" version, I believe (I may be incorrect) that there was miscommunication on their end since so many people are involved and that it's possible they haven't actually yet properly set up the networking on their end. The person assisting us with the IP announcements did check and showed it was configured on their router, however, some other things were said that makes me think maybe transit was never ready and we were just told everyone's ready and just waiting on IPv4 & racking. When I told them plans to do it sooner to get it up, by flying into San Jose, no one really said "no, the network drop isn't ready." So I don't know. It could of course be configuration issue on our end but we couldn't even ping their gateway or get any type of response.
They abruptly stopped replying to me at 1:44PM after otherwise communicating pretty quickly, so I don't know what happened. I'm trying to reach their support department now so it goes to someone who is available instead of email chains with very specific people that maybe work eastern time and left at 2PM (just a guess?)
Edit -- I do want to add something I remember specifically asking/mentioning, something along the lines of "okay, so if I fly to San Jose, to try to get customers up sooner, that is possible, right?" And I basically got a confirmation. I'm really hoping it's just my screw-up and they confirm everything so we at least have a chance of fixing it, otherwise it seems like it'll go into a communication loop on their end and take until Friday/Saturday.
Happy to see that my Dallas VMs that were all on the same server are now spread out on multiple NYC servers. They are all offline though. SolusVM seems to go through the motions of doing stuff, like telling me the servers booted successfully --- but they dont.
@JoeMerit said:
Happy to see that my Dallas VMs that were all on the same server are now spread out on multiple NYC servers. They are all offline though. SolusVM seems to go through the motions of doing stuff, like telling me the servers booted successfully --- but they dont.
When you go into solusVM does it show a blank under "Operating System" on the first page ?
Does it give you an "Authentication failed" error message when you try to use VNC ?
If so, you are not alone.
What nodes are you on in NYC that does this ?
Just to let VirMach know if this issue is affecting some nodes and not others.
I just saw this on the OGF and though it might be of interest to VirMach and others.
IPXO said:
Hey everyone!
Thank you for bringing your concerns to our attention. We welcome the opportunity to clarify our policies and procedures.
Firstly, there seems to be a misunderstanding regarding account suspension. Suspension does not impact your existing services or leased IP addresses. Instead, it temporarily restricts your ability to purchase new services until the underlying issue has been resolved.
We are committed to maintaining a high standard of integrity for the IP addresses available through our marketplace. However, while we remain vigilant against abuse, we also understand mishaps can occur. That's why we allow a tolerance of two incidents per client each month. If a client exceeds the tolerance threshold, we apply an incident handling fee. You can read more about our abuse handling policy here: https://www.ipxo.com/kb/abuse-prevention-and-reputation/abuse-incident-handling-marketplace/
Primary Reasons for Account Suspension:
If at least one subnet is listed in SpamHaus SBL account will be suspended until the issue is fully resolved.
Hijacking a subnet for more than 48H with repetitive illegal announcements.
Abuse reports escalated by IP owners, companies, or users (e.g., SPAM, phishing, attacks, copyright infringements).
Recurring abuse despite client's confirmation of resolution.
Primary Reasons for Account Termination:
If a client confirms issue resolution but abuse keeps reoccurring, and no response is received to our abuse reports within 24 hours, we may terminate the offending subnet. However, before taking the action to terminate we always check the correspondence with client, what actions were taken, and how the issue is/was resolved.
If we detect signs of spamming activities and spammer setup indications and the client fails to address the issue for a prolonged time or is unresponsive.
No termination is carried out without careful consideration of all positions, if the client is responsive, we terminate only in the exceptional cases.
IPXO’s purpose is to fill the market void by providing a platform where businesses can come and lease IP addresses for a defined period. Additionally, it serves as the primary destination for IP holders who wish to monetize their addresses. We are committed to assisting our clients in finding the essential tools and resources for effective IP management. If you encounter any issues or have concerns, our expert team is always available to help!
As far as I know the current migrations are due to the Dedipath closure. The Tampa (Hivelocity) and Chicago DC1 (Quadranet) and Chicago DC2 (Hivelocity) were unaffected by the Dedipath closure.
My VPS on node "NYCB021" in offline status. All control options are available on SolusVM panel. When I try to 'boot' the VPS, the panel confirm: "Success!", but after that ,the VPS status remain in 'offline' status on SolusVM and WHMCS too. The new IP address looks like working (Probably the VPS needs only a 'Reconfigure Networking' process - but cant do it, because it offline).
Anyone have experience VPS's from this node (NYCB021)? I not seen any information from this node (or just made a mistake and not found it)? Thank You! PS: Ther was not mounted any ISO, but clicked the 'Unmount' button.
@VirMach I noticed what appears to be an inconsistency on your web site for Premium Ryzen VPS, on this page,
I have marked the inconsistency on this screenshot.
I expect it is not a high priority right now, but I figured you might want to know about it.
@FrankZ said: @titus see my comment here. If it applies to you, VirMach is aware, see his comment here.
Thank You very much, @FrankZ !
Yes, this VPS location was previously - if i'm not wrong - in Dallas (node name: RYZE.DAL-Z003.VMS)
Yes, the VNC provide the "Authentication failed" error message.
Yes, the "Operating System" filed in the SolusVM panel is empty (but I have some VPS from different locations where this field is empty more than a year ago, and my VPS is working fine :$)
The currently node name is: "NYCB021" (on SolusVM)
They've had VPS for years now, they're just being deceptive about their marketing.
We currently do not offer or have any VPS solution outside our beta program. Our VPS offerings will be out of beta sometime in early October. We did acquire a VPS brand a few years ago as part of an acquisition. However, that solution was not easily integrated into Hivelocity's backend so we sold it off. The VPS solution we are introducing in a few weeks will be fully integrated and our customers will be able to deploy both VPS and bare metal from a single interface while also being able to create VLANS connecting both via our private network.
They've had VPS for years now, they're just being deceptive about their marketing.
We currently do not offer or have any VPS solution outside our beta program. Our VPS offerings will be out of beta sometime in early October. We did acquire a VPS brand a few years ago as part of an acquisition. However, that solution was not easily integrated into Hivelocity's backend so we sold it off. The VPS solution we are introducing in a few weeks will be fully integrated and our customers will be able to deploy both VPS and bare metal from a single interface while also being able to create VLANS connecting both via our private network.
Yeah I was thinking about SpeedyKVM, it never felt well integrated into the rest of the brand. I'm glad to see that the new solution is more tightly integrated, opens up possibilities for (somewhat) autoscaling clusters with metal and VPS. Excited to see what ends up being possible!
Is anyone on DALZ004? My Virmach Dallas VPS went offline at 16:17 UTC on the 11th. Virmach control panel gives me "Operation Timed Out After 90000 Milliseconds With 0 Bytes Received", so I'm assuming the host node is down.
But here's the strange bit--I'm seeing tons more DNS traffic than usual at my DNS server VPS (which isn't at Virmach). When I dig in and look at the DNS logs, I'm seeing lots of cloudflare requests for domains such as drive.shadow.tech.example.com, webdav.4shared.com.example.com, gateway.us1.storjshare.io.example.com, and axznbomgbocw.compat.objectstorage.us-phoenix-1.oraclecloud.com.example.com. (I've replaced my hostname with example.com for these examples.) Most are for A or AAAA records.
What scares me is that my Virmach Dallas VPS has an hourly borgbackup script that uses rclone to push the borg repo to offsite cloud storage, and those the cloudflare hostname lookups correspond with the cloud storage services, with my hostname appended to the end of the query (so they're all NXDOMAIN).
It looks like the massive DNS flood began with gateway.us1.storjshare.io.example.com at 2023-09-11 17:01:38 UTC.
I'm worried that something has been hacked, but I'm not sure what.
Thoughts, comments, suggestions, please!
Edit: is it possible my Virmach VPS is running and I just don't have access yet? I can't recall what DNS resolvers it was set to use, but maybe it's appending my hostname to failing queries and that's what's causing all the DNS traffic?
Edit again: I've pulled the last borg backup and checked my /etc/resolv.conf: #Generated by SolusVM nameserver 1.1.1.1 nameserver 8.8.8.8 nameserver 8.8.4.4
But if my VPS is running, perhaps with no network access, why am I seeing all the DNS queries? Maybe it's running and the host firewall allows the VPS to send, but not receive?
San Jose confirmed an issue on their end, which was fixed. I'll have more news soon, it might require that I fly in again or we use remote hands since I spent a lot of time trying to get networking up which didn't leave enough time for verifying all the other connections or setting up IPMI.
It should have worked when it went up, but didn't so it might mean the ports were flipped, I never got to have the facility check how they were previously plugged in.
Phoenix, looks like announcement issue so far, and IPMI issue as well. Already notified them.
You Dallas VPS is probably in NYC now. After you get the timeout in the billing panel, go to the left menu and click "management". This should open a larger left side menu. Click "Full VPS control panel" this should take you to the solusVM panel where you can see what node you are on, and confirm, or not, that your VM has been migrated to NYC and what your new IP is, if it has been moved.
Your DNS issue may have various reasons, but I expect it may be related to being assigned a new IP and your firewall not being setup for the new IP, or that your resolv.conf was reset when the networking was fixed for the new IP, but that is just a wild guess.
**Regarding Dallas in general* - a small sort-of update, not specifically for anyone facing issues, just general thoughts on that location overall.
I've reached out to a lot of Dallas datacenter, one thing seems to be the general consensus: setup fees are high, and nothing's ready with anyone until at least the end of the month. Dallas was historically a good location for cheap power, that doesn't really seem to be the case anymore either. Otherwise most people that were in Dallas just were there because they kind of got "forced" into it with a good special. There are people that actually wanted that location because they were nearby, but they were a small percentage of customers there, otherwise it was a relatively unpopular location.
I've also reached out to providers in Oklahoma City.
I've compared the realistic option in OKC and compared it with DAL, and here's what I discovered, in terms of latency.
OKC in this comparison is single-homed with Hurricane Electric, we'll probably proceed with HE + Lumen mix. OKC is also cheap enough to where we could likely do the two carriers, but even with single-home HE, it's not that terrible (which is surprising.) I made this list relatively quickly so it could be wrong for a few locations.
OKC is faster to:
Montreal
Vancouver
Fremont
Chicago
Spain
UK
Paris
NL
Norway
Turkey
UAE
India
Australia
Slower to:
LAX
SF
Denver
NYC
Egypt
Tokyo
Israel
Singapore
I think the added latency to some of the locations above could be fixed once we add in Lumen. For China, it mostly looks like RNG. One might be faster to a certain datacenter there, with higher packet loss, other might be slower with lower loss, and so on, but generally speaking OKC is better to a few places, and worse for most of them but it's all over the place so I didn't include it above.
The big bonus with OKC is that I technically live there, and I could be there for setup and everything else on short notice versus Dallas. I also have many trusted people in OKC that could help out. It could be a good location shift away from Dallas to make things more interesting, and since we're an OKC company it could just generally be cool to finally have colocation there.
Let me know what you guys think. I wish there were better carriers, but it really does seem like our two best options are HE and Lumen (CenturyLink.) There's some local providers that would make the connection better maybe in the vicinity but they're very limited in reach. There's also Cogent, but I refuse to go with Cogent.
You Dallas VPS is probably in NYC now. After you get the timeout in the billing panel, go to the left menu and click "management".
Thanks for the pointer, this is very helpful. Turns out it still shows me on RYZE.DAL-Z004.VMS with my old IP. VNC wouldn't connect. Can't power it off, either.
You Dallas VPS is probably in NYC now. After you get the timeout in the billing panel, go to the left menu and click "management".
Thanks for the pointer, this is very helpful. Turns out it still shows me on RYZE.DAL-Z004.VMS with my old IP. VNC wouldn't connect. Can't power it off, either.
DALZ004 ran into issues. We might have to get people up for that and maybe DALZ007 as new services without backup for now.
@VirMach OKC. I know you said no more dedicated servers. I know it's too crazy to imagine me sticking a server in together with your OKC servers. I dunno. I want a US server sometime before too long. Any ideas for me? It's okay not to answer if you are busy or don't want to for any reason. I am impressed by how many guys really, really like and trust you. So that's why I wanted to ask. Good luck! Better days ahead soon!
You Dallas VPS is probably in NYC now. After you get the timeout in the billing panel, go to the left menu and click "management".
Thanks for the pointer, this is very helpful. Turns out it still shows me on RYZE.DAL-Z004.VMS with my old IP. VNC wouldn't connect. Can't power it off, either.
DALZ004 ran into issues. We might have to get people up for that and maybe DALZ007 as new services without backup for now.
The VPS being down isn't costing me $millions/hour, so please don't take me comments as complaints. My main concern now is slowing this DNS storm. It is a very interesting situation, as though the VPS can send traffic but not receive any.
I'd rather it move to OKC than NYC. OKC vs. Dallas is a close one; I'm not sure it matters to me.
@Not_Oles said: @VirMach OKC. I know you said no more dedicated servers. I know it's too crazy to imagine me sticking a server in together with your OKC servers. I dunno. I want a US server sometime before too long. Any ideas for me? It's okay not to answer if you are busy or don't want to for any reason. I am impressed by how many guys really, really like and trust you. So that's why I wanted to ask. Good luck! Better days ahead soon!
I'm actually leaning toward going with a lot of space in OKC and specifically using that location to focus on our dedicated servers product as it would make the most sense.
Power is cheap there, and the more data we can get with many carriers the better. It ends up making way more sense than paying high setup fee and monthly fee for a cabinet as they have a ton of space there, just nothing already set up. So if we get a lot of power, space, and then just have it to set up cabinets and fill them out as we go and have a bunch of transit ready to go as we spin up new ones. So yeah, offering dedicated servers and even colocation is very realistic for that location.
For colocation we could offer, it would end up being in the ballpark of 1.6kW / 10U / 500Mbps 95th for $300 a month right now. That'd be with no setup fee and free initial scheduled setup and trying to be conservative, realistically it could be lower but with some setup fees. This is all in the early stages though so I don't know if we could get anything going for you any time this month.
Properly planned out date for launch would be some time November, if everything goes well with everything else it could maybe be near end of October but I highly doubt that.
Additional possibly good news for everyone else, based on the power quote I got we could end up being able to just put up all the old equipment we have gathering dust, that's E5's and E3's and just do some type of launch promotion for those at a good price, special only to kind of make amends for anyone who had dedicated servers with us over the last couple years and what they had to go through. Basically at-cost. The very early stages of this idea was to just buy out and do our own facility, I got power quotes from the power companies directly and quotes to run fiber but this was several years ago and it seems like a good company has already popped up and done all that for that area. Long ago, even before the las time I explored this seriously, it was pretty difficult getting anything done for any company in that area. Think like 2015. We got some insane quote from the only provider in that area, but more have popped up now. Yay, competition.
@VirMach said: The big bonus with OKC is that I technically live there, and I could be there for setup and everything else on short notice versus Dallas. I also have many trusted people in OKC that could help out. It could be a good location shift away from Dallas to make things more interesting, and since we're an OKC company it could just generally be cool to finally have colocation there.
This sounds like it would be a big plus for you. Truth be told I was picking up some weird routing with DediPath Dallas that was making other providers in Dallas 20 to 30 ms faster then my VMs with you in Dallas.
I'm good with OKC as I expect we might be able to get IPv6 there, right ?
@VirMach Thanks for your reply! OKC all sounds very interesting and good! For me, it's no rush, a few months doesn't matter. I will continue to watch your thread. You should feel free to PM me if you wish. Or to use the email on my profile. Thanks again!
San Jose is looking better than it did a few hours ago. It's possible nothing further needs to do be done and we're just waiting for announcement to propagate since it came on later than anticipated.
Could end up being online early as tonight or tomorrow morning. If it does require more to be done though then it's looking more like after tomorrow afternoon.
@FrankZ said: This sounds like it would be a big plus for you. Truth be told I was picking up some weird routing with DediPath Dallas that was making other providers in Dallas 20 to 30 ms faster then my VMs with you in Dallas.
I'm good with OKC as I expect we might be able to get IPv6 there, right ?
OKC we'd want to do as properly as possible. Yes that even means IPv6, our own ASN, and likely it'd go under a different brand with a different control panel and maybe even different server types (Epyc?) from the start for VPS. Of course it's possible VirMach could end up sub-leasing some space from this brand so I wouldn't know about what VirMach's going to do for their VPS offerings.
Routing I compared was DataPacket Dallas (CDN77) with OKC single-homed Hurricane Electric.
@FrankZ said: This sounds like it would be a big plus for you. Truth be told I was picking up some weird routing with DediPath Dallas that was making other providers in Dallas 20 to 30 ms faster then my VMs with you in Dallas.
I'm good with OKC as I expect we might be able to get IPv6 there, right ?
OKC we'd want to do as properly as possible. Yes that even means IPv6, our own ASN, and likely it'd go under a different brand with a different control panel and maybe even different server types (Epyc?) from the start for VPS. Of course it's possible VirMach could end up sub-leasing some space from this brand so I wouldn't know about what VirMach's going to do for their VPS offerings.
Routing I compared was DataPacket Dallas (CDN77) with OKC single-homed Hurricane Electric.
Highly recommend VirtFusion for control panel. It is prem.
@VirMach my seattle vps is down for a few weeks now it is shown as node locked I cant seem to get more details , I am not losing millions but any idea when this will be back? PS. I have zero tickets open for this as I dont want to bother you just wanted to ask casually if entire SEA is down or is it just me this was down before dedipath thing happened
@codelock said: @VirMach my seattle vps is down for a few weeks now it is shown as node locked I cant seem to get more details , I am not losing millions but any idea when this will be back? PS. I have zero tickets open for this as I dont want to bother you just wanted to ask casually if entire SEA is down or is it just me this was down before dedipath thing happened
Check network status page for the most important details right now.
@FrankZ said: @VirMach I noticed what appears to be an inconsistency on your web site for Premium Ryzen VPS, on this page,
I have marked the inconsistency on this screenshot.
I expect it is not a high priority right now, but I figured you might want to know about it.
We should fix it by making it give you less disk unless you scroll down.
@FrankZ said: @VirMach I noticed what appears to be an inconsistency on your web site for Premium Ryzen VPS, on this page,
I have marked the inconsistency on this screenshot.
I expect it is not a high priority right now, but I figured you might want to know about it.
We should fix it by making it give you less disk unless you scroll down.
LOL. You seem to be in good humor tonight. Or is that just me misinterpreting your lack of sleep, and excess caffeine intake ?
You Dallas VPS is probably in NYC now. After you get the timeout in the billing panel, go to the left menu and click "management".
Thanks for the pointer, this is very helpful. Turns out it still shows me on RYZE.DAL-Z004.VMS with my old IP. VNC wouldn't connect. Can't power it off, either.
DALZ004 ran into issues. We might have to get people up for that and maybe DALZ007 as new services without backup for now.
I'm on DALZ004 and have my own backups. I'd love the option to be moved to NYC as a new service, if possible.
Comments
Phoenix has been down for two days.
Wednesday is now over, if we consider normal business hours.
Will we get SLA credits for $0.00/month service?
Asking for a friend.
We accept Karma donations for the last flan. 🍮 affbrr
May I ask if DALZ007 is finally moving to NYC?
NYC is a terrible location for me, is there any possibility of staying in Dallas or moving back to Dallas in the future?
Does it show an O/S in solusVM, or just the billing panel ?
Read what VirMach said about this here
@Frankz read what @tulipyun wrote again please.
The Yeti has left the building.
A few comments up I said...
The VirMach comments that I linked to were interpreted by me as saying that Dallas is not going to be a VirMach location, at least in the near term, and it was unknown if that location would come back in the longer term.
What am I missing ?
My bad I need to stop and go do something else. I actually thought he said Dallas was bad for him and could it be moved to Dallas.
I do not like myself at the moment. I need to go to bed and forget everything for about 15-16 hours. I think I may have blew the ram right out of my brain.
The Yeti has left the building.
Just remember that we like you, even when you might not like yourself.
I have no issue with what you said , and honestly thought I was the one missing something.
Of course getting some sleep is never a bad thing, given that you have been hitting it pretty hard for some time now.
(according to my stalker cam)
Yeah it's there also
If I try VNC it show: Authentication failed...
Interesting, I also get "Authentication failed", both before and after changing the password for VNC, for my broken migrated VM on NYCB45. This seems to be effecting more than just a couple of VMs so I would expect that the VirMach crew is aware and looking for a solution.
Yeah Virmach got me again. This is brutal anymore. Time to go with the big boys for security I suppose.
The Yeti has left the building.
NYC connectivity should be up for the new blocks/migrations now. I'm just checking it further and will then update the network status page. Phoenix has been racked and should be ready, but BGP hasn't updated yet, I'm waiting to see if it slowly starts to show signs of life but I won't be able to check it any other way as I believe IPMI is still waiting.
San Jose... very long story. "Short" version, I believe (I may be incorrect) that there was miscommunication on their end since so many people are involved and that it's possible they haven't actually yet properly set up the networking on their end. The person assisting us with the IP announcements did check and showed it was configured on their router, however, some other things were said that makes me think maybe transit was never ready and we were just told everyone's ready and just waiting on IPv4 & racking. When I told them plans to do it sooner to get it up, by flying into San Jose, no one really said "no, the network drop isn't ready." So I don't know. It could of course be configuration issue on our end but we couldn't even ping their gateway or get any type of response.
They abruptly stopped replying to me at 1:44PM after otherwise communicating pretty quickly, so I don't know what happened. I'm trying to reach their support department now so it goes to someone who is available instead of email chains with very specific people that maybe work eastern time and left at 2PM (just a guess?)
Edit -- I do want to add something I remember specifically asking/mentioning, something along the lines of "okay, so if I fly to San Jose, to try to get customers up sooner, that is possible, right?" And I basically got a confirmation. I'm really hoping it's just my screw-up and they confirm everything so we at least have a chance of fixing it, otherwise it seems like it'll go into a communication loop on their end and take until Friday/Saturday.
@VirMach, pls confirm whether DALZ007 moved to N.Y or stays.
Also Migrating from Seattle to LAX is on hold or going forward?
https://lowendspirit.com/discussion/comment/150211/#Comment_150211
DAL is going bye bye until further notice.
Happy to see that my Dallas VMs that were all on the same server are now spread out on multiple NYC servers. They are all offline though. SolusVM seems to go through the motions of doing stuff, like telling me the servers booted successfully --- but they dont.
When you go into solusVM does it show a blank under "Operating System" on the first page ?
Does it give you an "Authentication failed" error message when you try to use VNC ?
If so, you are not alone.
What nodes are you on in NYC that does this ?
Just to let VirMach know if this issue is affecting some nodes and not others.
I just saw this on the OGF and though it might be of interest to VirMach and others.
Btw... Tampa and Chicago are "fine" or we should think about migrations too?
As far as I know the current migrations are due to the Dedipath closure. The Tampa (Hivelocity) and Chicago DC1 (Quadranet) and Chicago DC2 (Hivelocity) were unaffected by the Dedipath closure.
My VPS on node "NYCB021" in offline status. All control options are available on SolusVM panel. When I try to 'boot' the VPS, the panel confirm: "Success!", but after that ,the VPS status remain in 'offline' status on SolusVM and WHMCS too. The new IP address looks like working (Probably the VPS needs only a 'Reconfigure Networking' process - but cant do it, because it offline).
Anyone have experience VPS's from this node (NYCB021)? I not seen any information from this node (or just made a mistake and not found it)? Thank You! PS: Ther was not mounted any ISO, but clicked the 'Unmount' button.
@VirMach I noticed what appears to be an inconsistency on your web site for Premium Ryzen VPS, on this page,
I have marked the inconsistency on this screenshot.
I expect it is not a high priority right now, but I figured you might want to know about it.
@titus see my comment here. If it applies to you, VirMach is aware, see his comment here.
Thank You very much, @FrankZ !
Yes, this VPS location was previously - if i'm not wrong - in Dallas (node name: RYZE.DAL-Z003.VMS)
Yes, the VNC provide the "Authentication failed" error message.
Yes, the "Operating System" filed in the SolusVM panel is empty (but I have some VPS from different locations where this field is empty more than a year ago, and my VPS is working fine :$)
The currently node name is: "NYCB021" (on SolusVM)
We currently do not offer or have any VPS solution outside our beta program. Our VPS offerings will be out of beta sometime in early October. We did acquire a VPS brand a few years ago as part of an acquisition. However, that solution was not easily integrated into Hivelocity's backend so we sold it off. The VPS solution we are introducing in a few weeks will be fully integrated and our customers will be able to deploy both VPS and bare metal from a single interface while also being able to create VLANS connecting both via our private network.
Yeah I was thinking about SpeedyKVM, it never felt well integrated into the rest of the brand. I'm glad to see that the new solution is more tightly integrated, opens up possibilities for (somewhat) autoscaling clusters with metal and VPS. Excited to see what ends up being possible!
Is anyone on DALZ004? My Virmach Dallas VPS went offline at 16:17 UTC on the 11th. Virmach control panel gives me "Operation Timed Out After 90000 Milliseconds With 0 Bytes Received", so I'm assuming the host node is down.
But here's the strange bit--I'm seeing tons more DNS traffic than usual at my DNS server VPS (which isn't at Virmach). When I dig in and look at the DNS logs, I'm seeing lots of cloudflare requests for domains such as drive.shadow.tech.example.com, webdav.4shared.com.example.com, gateway.us1.storjshare.io.example.com, and axznbomgbocw.compat.objectstorage.us-phoenix-1.oraclecloud.com.example.com. (I've replaced my hostname with example.com for these examples.) Most are for A or AAAA records.
What scares me is that my Virmach Dallas VPS has an hourly borgbackup script that uses rclone to push the borg repo to offsite cloud storage, and those the cloudflare hostname lookups correspond with the cloud storage services, with my hostname appended to the end of the query (so they're all NXDOMAIN).
It looks like the massive DNS flood began with gateway.us1.storjshare.io.example.com at 2023-09-11 17:01:38 UTC.
WTF? Surely someone didn't guess axznbomgbocw.compat.objectstorage.us-phoenix-1.oraclecloud.com.
I'm worried that something has been hacked, but I'm not sure what.
Thoughts, comments, suggestions, please!
Edit: is it possible my Virmach VPS is running and I just don't have access yet? I can't recall what DNS resolvers it was set to use, but maybe it's appending my hostname to failing queries and that's what's causing all the DNS traffic?
Edit again: I've pulled the last borg backup and checked my /etc/resolv.conf:
#Generated by SolusVM nameserver 1.1.1.1 nameserver 8.8.8.8 nameserver 8.8.4.4But if my VPS is running, perhaps with no network access, why am I seeing all the DNS queries? Maybe it's running and the host firewall allows the VPS to send, but not receive?
San Jose confirmed an issue on their end, which was fixed. I'll have more news soon, it might require that I fly in again or we use remote hands since I spent a lot of time trying to get networking up which didn't leave enough time for verifying all the other connections or setting up IPMI.
It should have worked when it went up, but didn't so it might mean the ports were flipped, I never got to have the facility check how they were previously plugged in.
Phoenix, looks like announcement issue so far, and IPMI issue as well. Already notified them.
You Dallas VPS is probably in NYC now. After you get the timeout in the billing panel, go to the left menu and click "management". This should open a larger left side menu. Click "Full VPS control panel" this should take you to the solusVM panel where you can see what node you are on, and confirm, or not, that your VM has been migrated to NYC and what your new IP is, if it has been moved.
Your DNS issue may have various reasons, but I expect it may be related to being assigned a new IP and your firewall not being setup for the new IP, or that your resolv.conf was reset when the networking was fixed for the new IP, but that is just a wild guess.
**Regarding Dallas in general* - a small sort-of update, not specifically for anyone facing issues, just general thoughts on that location overall.
I've reached out to a lot of Dallas datacenter, one thing seems to be the general consensus: setup fees are high, and nothing's ready with anyone until at least the end of the month. Dallas was historically a good location for cheap power, that doesn't really seem to be the case anymore either. Otherwise most people that were in Dallas just were there because they kind of got "forced" into it with a good special. There are people that actually wanted that location because they were nearby, but they were a small percentage of customers there, otherwise it was a relatively unpopular location.
I've also reached out to providers in Oklahoma City.
I've compared the realistic option in OKC and compared it with DAL, and here's what I discovered, in terms of latency.
OKC in this comparison is single-homed with Hurricane Electric, we'll probably proceed with HE + Lumen mix. OKC is also cheap enough to where we could likely do the two carriers, but even with single-home HE, it's not that terrible (which is surprising.) I made this list relatively quickly so it could be wrong for a few locations.
OKC is faster to:
Slower to:
I think the added latency to some of the locations above could be fixed once we add in Lumen. For China, it mostly looks like RNG. One might be faster to a certain datacenter there, with higher packet loss, other might be slower with lower loss, and so on, but generally speaking OKC is better to a few places, and worse for most of them but it's all over the place so I didn't include it above.
The big bonus with OKC is that I technically live there, and I could be there for setup and everything else on short notice versus Dallas. I also have many trusted people in OKC that could help out. It could be a good location shift away from Dallas to make things more interesting, and since we're an OKC company it could just generally be cool to finally have colocation there.
Let me know what you guys think. I wish there were better carriers, but it really does seem like our two best options are HE and Lumen (CenturyLink.) There's some local providers that would make the connection better maybe in the vicinity but they're very limited in reach. There's also Cogent, but I refuse to go with Cogent.
@VirMach, good idea, like OKC. If My Dallas vm can move to OKC, I will wait a month.
Thanks for the pointer, this is very helpful. Turns out it still shows me on RYZE.DAL-Z004.VMS with my old IP. VNC wouldn't connect. Can't power it off, either.
DALZ004 ran into issues. We might have to get people up for that and maybe DALZ007 as new services without backup for now.
Just don't move the Dallas servers to NYC.
So OKC is a good choice
I can't wait one month with two Dallas VPS offline ( have some system working there ) ... I hope I could power on NY and migrate in future
@VirMach OKC. I know you said no more dedicated servers. I know it's too crazy to imagine me sticking a server in together with your OKC servers. I dunno. I want a US server sometime before too long. Any ideas for me? It's okay not to answer if you are busy or don't want to for any reason. I am impressed by how many guys really, really like and trust you. So that's why I wanted to ask. Good luck! Better days ahead soon!
The VPS being down isn't costing me $millions/hour, so please don't take me comments as complaints. My main concern now is slowing this DNS storm. It is a very interesting situation, as though the VPS can send traffic but not receive any.
I'd rather it move to OKC than NYC. OKC vs. Dallas is a close one; I'm not sure it matters to me.
BTW, here's a graph showing the fun: https://imgur.com/nuNc8Qb
I'm actually leaning toward going with a lot of space in OKC and specifically using that location to focus on our dedicated servers product as it would make the most sense.
Power is cheap there, and the more data we can get with many carriers the better. It ends up making way more sense than paying high setup fee and monthly fee for a cabinet as they have a ton of space there, just nothing already set up. So if we get a lot of power, space, and then just have it to set up cabinets and fill them out as we go and have a bunch of transit ready to go as we spin up new ones. So yeah, offering dedicated servers and even colocation is very realistic for that location.
For colocation we could offer, it would end up being in the ballpark of 1.6kW / 10U / 500Mbps 95th for $300 a month right now. That'd be with no setup fee and free initial scheduled setup and trying to be conservative, realistically it could be lower but with some setup fees. This is all in the early stages though so I don't know if we could get anything going for you any time this month.
Properly planned out date for launch would be some time November, if everything goes well with everything else it could maybe be near end of October but I highly doubt that.
Additional possibly good news for everyone else, based on the power quote I got we could end up being able to just put up all the old equipment we have gathering dust, that's E5's and E3's and just do some type of launch promotion for those at a good price, special only to kind of make amends for anyone who had dedicated servers with us over the last couple years and what they had to go through. Basically at-cost. The very early stages of this idea was to just buy out and do our own facility, I got power quotes from the power companies directly and quotes to run fiber but this was several years ago and it seems like a good company has already popped up and done all that for that area. Long ago, even before the las time I explored this seriously, it was pretty difficult getting anything done for any company in that area. Think like 2015. We got some insane quote from the only provider in that area, but more have popped up now. Yay, competition.
rooting for san joseeeeeeeeeee
I bench YABS 24/7/365 unless it's a leap year.
This sounds like it would be a big plus for you. Truth be told I was picking up some weird routing with DediPath Dallas that was making other providers in Dallas 20 to 30 ms faster then my VMs with you in Dallas.
I'm good with OKC as I expect we might be able to get IPv6 there, right ?
@VirMach Thanks for your reply! OKC all sounds very interesting and good! For me, it's no rush, a few months doesn't matter. I will continue to watch your thread. You should feel free to PM me if you wish. Or to use the email on my profile. Thanks again!
San Jose is looking better than it did a few hours ago. It's possible nothing further needs to do be done and we're just waiting for announcement to propagate since it came on later than anticipated.
Could end up being online early as tonight or tomorrow morning. If it does require more to be done though then it's looking more like after tomorrow afternoon.
Showing signs of life:
OKC we'd want to do as properly as possible. Yes that even means IPv6, our own ASN, and likely it'd go under a different brand with a different control panel and maybe even different server types (Epyc?) from the start for VPS. Of course it's possible VirMach could end up sub-leasing some space from this brand so I wouldn't know about what VirMach's going to do for their VPS offerings.
Routing I compared was DataPacket Dallas (CDN77) with OKC single-homed Hurricane Electric.
Highly recommend VirtFusion for control panel. It is prem.
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
@VirMach my seattle vps is down for a few weeks now it is shown as node locked I cant seem to get more details , I am not losing millions but any idea when this will be back? PS. I have zero tickets open for this as I dont want to bother you just wanted to ask casually if entire SEA is down or is it just me this was down before dedipath thing happened
Want free vps ? https://microlxc.net
Check network status page for the most important details right now.
We should fix it by making it give you less disk unless you scroll down.
If Dallas stuff was permanently moved to anywhere other than Dallas, I would not renew those ones.
LOL. You seem to be in good humor tonight. Or is that just me misinterpreting your lack of sleep, and excess caffeine intake ?
I'm on DALZ004 and have my own backups. I'd love the option to be moved to NYC as a new service, if possible.