@willie said:
How does an IP address get stolen from a VPS, except through someone getting to the control panel somehow?
SolusVM bug in this case. There's a lot of different bugs. This instance happened to have more than one. Basically at any given time it's possible for it to skip one of the rules or assignments for whatever reason and it creates a chain reaction over time. There's another similar bug with disks that they never fixed, they just gave us a workaround which we haven't used as it's very I/O intensive and could kill the disk more quickly.
Some IP stealer also simultaneously took advantage of the situation, whether on purpose or not.
So there were essentially three people competing over one IP. How IP stealing usually can happen with the protection turned on is if one of the bugs above exists. Sometimes it also fully fails but that's more rare. It definitely didn't help that we also had to turn it off for some time when the NICs were burning up from the other issue we faced before splitting VLANs. And the short version of that goes: integrated NIC isn't amazing, so it struggles keeping up with so many ARP packets flying around, and once we turned on the protection it added another layer of things it had to keep track of which made it want to melt.
Fun stuff.
(edit) I suppose it's not necessarily a bug, it was just never coded to handle all the weird scenarios that may occur and doesn't clean up automatically. After all, SolusVM in its current state is someone used PHP where PHP probably shouldn't have been used and a bunch of low end hosts such as myself decided to use it anyway. Back in the 1800's when it was coded though maybe it was the only option.
Mentally strong people put each KVM in a separate VLAN.
Each IPv4 /32 or IPv6 /56 is individually routed to the VLAN.
Gateway address is a link-local address, for both IPv4 and IPv6.
IP stealing is impossible and no IP is wasted for gateway.
We accept Karma donations for the last flan. 🍮 affbrr
Thanks for sorting it out Virmach. Now where's that big hammer?
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@packer255 said:
Has a round of maintenance for TYOC040 ended? 72 hours have passed, but my vps is still offline.
TYOC040 Still updating the announcement, very good. TYOC035 have been offline for more than a month.
Status Offline
Node Name TYOC035
TYOC035 (Outage)Critical
Affecting Server - SolusVM
09/13/2022 16:39 Last Updated 09/13/2022 16:39
This issue affects a server that may impact your services
TYOC035 is facing a partial outage. We are evaluating a potential maintenance window.
@packer255 said:
Has a round of maintenance for TYOC040 ended? 72 hours have passed, but my vps is still offline.
TYOC040 Still updating the announcement, very good. TYOC035 have been offline for more than a month.
Status Offline
Node Name TYOC035
TYOC035 (Outage)Critical
Affecting Server - SolusVM
09/13/2022 16:39 Last Updated 09/13/2022 16:39
This issue affects a server that may impact your services
TYOC035 is facing a partial outage. We are evaluating a potential maintenance window.
You know I've thought long and hard about it and decided you were right, I was deceiving myself. I want you to know I appreciate you pointing this out.
i think virmach has to fix the MJJ Tokyo unorthodox migration influx in parallel with getting the nodes back up to have a better control of the situation.
You know I've thought long and hard about it and decided you were right, I was deceiving myself. I want you to know I appreciate you pointing this out.
@cybertech said:
i think virmach has to fix the MJJ Tokyo unorthodox migration influx in parallel with getting the nodes back up to have a better control of the situation.
You know I've thought long and hard about it and decided you were right, I was deceiving myself. I want you to know I appreciate you pointing this out.
Do u have any service in TYOC035?
No, and for that you have my sympathy. Truly unlucky for you.
You know I've thought long and hard about it and decided you were right, I was deceiving myself. I want you to know I appreciate you pointing this out.
Do u have any service in TYOC035?
No, and for that you have my sympathy. Truly unlucky for you.
No, and for that you have my sympathy. Truly unlucky for you.
So.This is why u can not understand.
I understand fine. I understand that you're unhappy/angry/upset. I understand you've paid for a service you aren't receiving. I understand this causes you great stress. I understand that you want to make all of these things known. The unresolved question is whether you understand that you're using words incorrectly. It's not abandoned. It's not ignored. It just hasn't gotten fixed yet because other things are being done first. Is that fair? (shrug)
Currently 22.5% of TYOC035 is potentially impacted.
Currently 40% of TYOC040 is potentially impacted.
That means 77.5% of TYOC035 isn't and 60% of TYOC040 isn't. Even with this, we've still been getting so many tickets about them when there's already a network status page posted and it can also be seen here on this thread with how often it's getting mentioned when in reality it's not impacting as many people as you'd assume when compared to other topics that have been discussed.
Right now SJCZ004 outage is impacting more people than both of those nodes combined. Tell me how many times that has been mentioned versus TYOC035 and TYOC040.
SEAZ009 also went down and it's impacting more people as well. How many mentions for that? We received infinite times more tickets about TYOC035 and TYOC040 (because it's 0, since people looked at the network status page for it.)
If we're already having a high support load as a result of just 35% of the people being affected, then by taking it offline to do further maintenance that's essentially guaranteed we'll receive 3x as many reports from 100% of the people being affected. Last time for TYOC040 we sent an email, made a network status, and even asked people to consider not contacting us so we can focus on the issue and we still got many requests. So while I understand it's clearly a big problem and I apologize for the continued outage for those people, understand that when we're being simultaneously bombarded with requests it's counterproductive and that we also have other issues we're tending to so it makes it a better choice to prioritize those at the moment when the only thing that's essentially guaranteed to come from another maintenance or migration attempt is a lot of tickets. Even if we migrate people off, we're going to end up getting a flood of tickets for that so we definitely do not want to continue having ticket backlogs and have to keep that in mind.
I've been in a bit of a predicament with VirMach for a while... For a start, I understand that VirMach has been having a lot of trouble lately with ColoCrossing - speaking of which, what happened to them? I have (or had) a dedicated server, which I haven't had access to for nearly 6 months because of a networking issue. Then last month, overnight, I got an email saying it was suddenly being "migrated" elsewhere, and that I should back up my data. Well this is the problem, I couldn't back it up, since I hadn't had access to it for months on end. Now I'm being told that the data is lost and there's little chance of it being recovered. So what happens now?
@MemphisX said:
I've been in a bit of a predicament with VirMach for a while... For a start, I understand that VirMach has been having a lot of trouble lately with ColoCrossing - speaking of which, what happened to them? I have (or had) a dedicated server, which I haven't had access to for nearly 6 months because of a networking issue. Then last month, overnight, I got an email saying it was suddenly being "migrated" elsewhere, and that I should back up my data. Well this is the problem, I couldn't back it up, since I hadn't had access to it for months on end. Now I'm being told that the data is lost and there's little chance of it being recovered. So what happens now?
We accept Karma donations for the last flan. 🍮 affbrr
I was affected too. Follow the instructions in the email (also copied in this comment: https://lowendspirit.com/discussion/comment/94512/#Comment_94512, additional info here: https://lowendspirit.com/discussion/comment/94592/#Comment_94592) to create a ticket to (hopefully) get a new service deployed quickly. A ticket should have been generated for you automatically, but maybe it got missed/buried and put on the back burner with everything else going on, so best to open that priority support ticket. Best of luck!
Ohh and yeah -- getting the data back isn't going to be an option at this point.... ColonCrossing made sure of that.
Disk is full on my Seafile instance.
The only machine with more disk is VirMach.
I spent all night moving the files.
Hopefully MIAZ011 doesn't lose data going forward.
Hourly backup is in place.
We accept Karma donations for the last flan. 🍮 affbrr
@Mason said: Ohh and yeah -- getting the data back isn't going to be an option at this point.... ColonCrossing made sure of that.
For dedicated servers too? Yikes.
Can someone tell me what MJJ is? Malaysian Jiu-Jitsu? I only know about the Brazilian kind.
If I understand the Tokyo stowaway thing, it sounds crazy that they haven't been booted, but it is also a serious software bug that it could happen at all.
I gotta add, both of my SJC vps are up now, have been up for the past week or two, and performing extremely well, as if the host nodes are nearly unloaded. Both have had IP changes and needed reinstalls but with that out of the way things are running smoothly. There may have been network issues that I haven't noticed since I've been idling both, after migrating a service away from one during the outage. I plan to bring that service back and see what happens.
@Virmach I know that you've said in the past that Transfers can be time consuming, but..
If you fancy a change from the "mundane" usual Tickets, could you please process the (merged) open Ticket for one.
Poor @vyas already paid me at the beginning of the month and it doesn't feel right holding onto his bucks, for nothing.
Go on, you know you want to.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
In other news, a Ryzen Migration from ATL to SEA went flawlessly.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Aye! A wee experiment, as I have a client in that neck of the woods.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I'm thinking that I could do with a second one there.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said:
Currently 22.5% of TYOC035 is potentially impacted.
Currently 40% of TYOC040 is potentially impacted.
That means 77.5% of TYOC035 isn't and 60% of TYOC040 isn't. Even with this, we've still been getting so many tickets about them when there's already a network status page posted and it can also be seen here on this thread with how often it's getting mentioned when in reality it's not impacting as many people as you'd assume when compared to other topics that have been discussed.
Right now SJCZ004 outage is impacting more people than both of those nodes combined. Tell me how many times that has been mentioned versus TYOC035 and TYOC040.
SEAZ009 also went down and it's impacting more people as well. How many mentions for that? We received infinite times more tickets about TYOC035 and TYOC040 (because it's 0, since people looked at the network status page for it.)
If we're already having a high support load as a result of just 35% of the people being affected, then by taking it offline to do further maintenance that's essentially guaranteed we'll receive 3x as many reports from 100% of the people being affected. Last time for TYOC040 we sent an email, made a network status, and even asked people to consider not contacting us so we can focus on the issue and we still got many requests. So while I understand it's clearly a big problem and I apologize for the continued outage for those people, understand that when we're being simultaneously bombarded with requests it's counterproductive and that we also have other issues we're tending to so it makes it a better choice to prioritize those at the moment when the only thing that's essentially guaranteed to come from another maintenance or migration attempt is a lot of tickets. Even if we migrate people off, we're going to end up getting a flood of tickets for that so we definitely do not want to continue having ticket backlogs and have to keep that in mind.
its trademark MJJ , even more so for unorthodox migrants.
whats the issue with node 40? will you be extending service for those affected instead of credits? credits cant help me get back 6TB of premium bandwidth watching educational videos in Japan.
@MemphisX said:
I've been in a bit of a predicament with VirMach for a while... For a start, I understand that VirMach has been having a lot of trouble lately with ColoCrossing - speaking of which, what happened to them? I have (or had) a dedicated server, which I haven't had access to for nearly 6 months because of a networking issue. Then last month, overnight, I got an email saying it was suddenly being "migrated" elsewhere, and that I should back up my data. Well this is the problem, I couldn't back it up, since I hadn't had access to it for months on end. Now I'm being told that the data is lost and there's little chance of it being recovered. So what happens now?
I'm not sure if you're speaking of us or ColoCrossing in your comment but if it's us then the timeline doesn't correlate and it's possible you're facing some other layer of issues.
@Mason said:
Ohh and yeah -- getting the data back isn't going to be an option at this point.... ColonCrossing made sure of that.
I believe we can share this information but I'll just say it based on what we had planned to potentially do rather than what we actually have done.
We wanted to give them a chance to consider powering these on, any servers that were paid for additionally through the end of August/September based on the vague communication we may have been provided regarding how they were going to proceed, whether or not it actually be what was supposed to happen. So that we could provide access to customers and reduce the damages caused. Since that did not happen you can make your own inferences on what happened.
@AlwaysSkint said: @Virmach I know that you've said in the past that Transfers can be time consuming, but..
If you fancy a change from the "mundane" usual Tickets, could you please process the (merged) open Ticket for one.
Poor @vyas already paid me at the beginning of the month and it doesn't feel right holding onto his bucks, for nothing.
Go on, you know you want to.
For probably years at this point we've been trying to streamline the process. It obviously hasn't happened. It's just very easy for these tickets to stay indefinitely in limbo because everyone other than myself is essentially afraid to do it since it involves a lot of security. As in, all the red tape I put over it basically puts the #1 priority at making sure it's impossible for a hacked account to lose their service or for the wrong account to go to the wrong place or for a SolusVM account with multiple services to accidentally be transferred instead of just the one service and so on. It's a lot of triple checking everything, all the logs on each account, sometimes pages of it, and so on.
I know some people might think we just do it to charge $3 and then profit and then just do nothing for weeks or months but that's not really the case.
@tenpera said: @Virmach I was glad to see ipv6 working at Germany, thanks.
I didn't do anything, sorry, lol. I'm actually surprised it works, it shouldn't but I guess it probably does on one node still. May I know which node it is so we can try to preserve it?
@AlwaysSkint said:
In other news, a Ryzen Migration from ATL to SEA went flawlessly.
did the same from LAX to AMS. now happy with xTom.
will Virmach plan to supply IPv6 in that location?
IPv6 will be in all locations except potentially some delay in Hivelocity. Their IPv4 pricing is crazy so I'm not sure how it's going to be for IPv6 but I can't imagine it'd be too terrible. We already have large blocks with everyone else. QN, I think has the smallest IPv6 blocks assigned to us. So it goes xTom --> Dedipath --> QN --> Hivelocity in terms of likelihood for problems with direct providers based on the quantity we have per cabinet I think right now.
The plan is to do 1x IPV6 and 1x /64 IPv6 subnet. Technically with some partners we have enough to where we could assign more than /64 IPv6 such as a /56 IPv6 but I think that's a good amount (/64 IPv6.) Open to suggestions.
We also want to set up a failover IPv4 NAT for every VPS as well. I'm trying to see if we can essentially purchase one IPv4 block for this so it's actually ours, owned by us, and do that with it so no matter what customers will never lose full connectivity just as a result of some third party deciding to pull IPv4. And then that essentially will also open the door to ultra low priced NAT offerings with IPv6. Think $7/YR instead of $12/YR with IPv4.
@cybertech said:
whats the issue with node 40? will you be extending service for those affected instead of credits? credits cant help me get back 6TB of premium bandwidth watching educational videos in Japan.
The issue will require several reboots and potentially hard power off and reseating. One of the disks fully disappeared and for some reason the system isn't throwing errors which makes... no sense really but I think I've seen it happen maybe once before a long time ago. Usually if a disk goes away in the OS it would throw errors saying it ever existed in the past and marked as [UNKNOWN] but this time it's not there and it's acting as if it was never there.
Sorry if that information sounds useless I just won't know more until we power it off to go into BIOS and check a few other things.
Credits, I don't know how we'll end up doing that. I don't want to get in another situation where we delay fixing it specifically to be able to go through manually and do any specific type of crediting because it becomes more difficult to track and credit after it's fixed especially if they move around.
IPv4 Update - We signed the contract with the new provider and working to get it done within our timeline. I know we're behind by a day so we'll push the notice into the change officially happening on Thursday instead of next Wednesday.
We've tested the first step and it looks to be going very smoothly. So you should get a ticket tonight with your exact IPv4 to expect.
@cybertech said:
whats the issue with node 40? will you be extending service for those affected instead of credits? credits cant help me get back 6TB of premium bandwidth watching educational videos in Japan.
The issue will require several reboots and potentially hard power off and reseating. One of the disks fully disappeared and for some reason the system isn't throwing errors which makes... no sense really but I think I've seen it happen maybe once before a long time ago. Usually if a disk goes away in the OS it would throw errors saying it ever existed in the past and marked as [UNKNOWN] but this time it's not there and it's acting as if it was never there.
Sorry if that information sounds useless I just won't know more until we power it off to go into BIOS and check a few other things.
Credits, I don't know how we'll end up doing that. I don't want to get in another situation where we delay fixing it specifically to be able to go through manually and do any specific type of crediting because it becomes more difficult to track and credit after it's fixed especially if they move around.
i understand from your point of view, but as a customer its not that fair either. the only consolation is that the service is lowendpriced.
anyway i dont think there's any effective solution to this apart from getting the nodes up and stable as a priority.
in any case hope your other battles are going strong.
@cybertech said:
whats the issue with node 40? will you be extending service for those affected instead of credits? credits cant help me get back 6TB of premium bandwidth watching educational videos in Japan.
The issue will require several reboots and potentially hard power off and reseating. One of the disks fully disappeared and for some reason the system isn't throwing errors which makes... no sense really but I think I've seen it happen maybe once before a long time ago. Usually if a disk goes away in the OS it would throw errors saying it ever existed in the past and marked as [UNKNOWN] but this time it's not there and it's acting as if it was never there.
Sorry if that information sounds useless I just won't know more until we power it off to go into BIOS and check a few other things.
Credits, I don't know how we'll end up doing that. I don't want to get in another situation where we delay fixing it specifically to be able to go through manually and do any specific type of crediting because it becomes more difficult to track and credit after it's fixed especially if they move around.
i understand from your point of view, but as a customer its not that fair either. the only consolation is that the service is lowendpriced.
anyway i dont think there's any effective solution to this apart from getting the nodes up and stable as a priority.
in any case hope your other battles are going strong.
We'll definitely get it figured out in terms of credit. I'm not saying we won't provide any, just that it may be done later so we can focus on the outage.
@AlwaysSkint said:
In other news, a Ryzen Migration from ATL to SEA went flawlessly.
did the same from LAX to AMS. now happy with xTom.
will Virmach plan to supply IPv6 in that location?
IPv6 will be in all locations except potentially some delay in Hivelocity. Their IPv4 pricing is crazy so I'm not sure how it's going to be for IPv6 but I can't imagine it'd be too terrible. We already have large blocks with everyone else. QN, I think has the smallest IPv6 blocks assigned to us. So it goes xTom --> Dedipath --> QN --> Hivelocity in terms of likelihood for problems with direct providers based on the quantity we have per cabinet I think right now.
The plan is to do 1x IPV6 and 1x /64 IPv6 subnet. Technically with some partners we have enough to where we could assign more than /64 IPv6 such as a /56 IPv6 but I think that's a good amount (/64 IPv6.) Open to suggestions.
We also want to set up a failover IPv4 NAT for every VPS as well. I'm trying to see if we can essentially purchase one IPv4 block for this so it's actually ours, owned by us, and do that with it so no matter what customers will never lose full connectivity just as a result of some third party deciding to pull IPv4. And then that essentially will also open the door to ultra low priced NAT offerings with IPv6. Think $7/YR instead of $12/YR with IPv4.
i just needed wanted 1xIPv6 so anything goes i guess?
although with /64 its a ready backup in case of any unforeseen IPv4 mishaps that may cripple services especially on a scale such as yours.
but NAT priced offerings? freaking awesome. it could be a win-win for true lowenders without provider bearing the liabilities of IPv4.
on that note i would suggest an "NAT port control panel" that can forward ports on the portal, instead of manually configuring it within the VPS. beginners like me wont have to sweat it then. i think this would fly.
having more control over IPs and Hardware, moving in positive direction amidst the teething issues.
Phoenix has been notified of IP address change. The emails have been sent from SolusVM. This location is not yet getting the new IP address. We're preparing to send in the tickets (emails will come from WHMCS) for the other locations. These will have the new IP address included with them.
IPv6 will be in all locations except potentially some delay in Hivelocity. Their IPv4 pricing is crazy so I'm not sure how it's going to be for IPv6 but I can't imagine it'd be too terrible. We already have large blocks with everyone else. QN, I think has the smallest IPv6 blocks assigned to us. So it goes xTom --> Dedipath --> QN --> Hivelocity in terms of likelihood for problems with direct providers based on the quantity we have per cabinet I think right now.
The plan is to do 1x IPV6 and 1x /64 IPv6 subnet. Technically with some partners we have enough to where we could assign more than /64 IPv6 such as a /56 IPv6 but I think that's a good amount (/64 IPv6.) Open to suggestions.
The plan was to originally provide 1 x /64 IPv6 but we decided to also provide a single IPv6 which both facilitates using it through SolusVM and is more user friendly. Adding in the 1x IPv6 is easier than adding the /64 IPv6 so a long time ago I believe I said that the /64 IPv6 may not be guaranteed but currently I don't see any reason why this wouldn't be possible.
@cybertech said: but NAT priced offerings? freaking awesome. it could be a win-win for true lowenders without provider bearing the liabilities of IPv4.
on that note i would suggest an "NAT port control panel" that can forward ports on the portal, instead of manually configuring it within the VPS. beginners like me wont have to sweat it then. i think this would fly.
Yeah we've been meaning to do this for quite some time but in the past with no IPv6 and the fact that SolusVM doesn't natively support it, we ended up not doing it. But once we finally get past all this craziness it shouldn't be too much of a headache to set up a script that configures it automatically and with IPv6 it's a completed solution.
@cybertech said:
having more control over IPs and Hardware, moving in positive direction amidst the teething issues.
Yeah, I'm very excited about this part of our company's lifetime.
@VirMach said:
Phoenix has been notified of IP address change. The emails have been sent from SolusVM. This location is not yet getting the new IP address. We're preparing to send in the tickets (emails will come from WHMCS) for the other locations. These will have the new IP address included with them.
@VirMach said: Hivelocity. Their IPv4 pricing is crazy so I'm not sure how it's going to be for IPv6 but I can't imagine it'd be too terrible
If you have an ASN, getting your own v6 space should be pretty easy still. It is v4 that is difficult.
@VirMach said: We also want to set up a failover IPv4 NAT for every VPS as well. I'm trying to see if we can essentially purchase one IPv4 block for this so it's actually ours, owned by us, and do that with it so no matter what customers will never lose full connectivity just as a result of some third party deciding to pull IPv4.
This doesn't sound worthwhile. v6-only is almost as good, in the event that the VPS's dedicated v4 address goes away. Whatever abuse was going on will likely take out the shared v4 address too. That happens all the time with NAT services. LES got its start that way and super cheap NAT VPS is fun, but was never very reliable because of the shared address being an abuse and DDOS target.
No, and for that you have my sympathy. Truly unlucky for you.
So.This is why u can not understand.
I understand fine. I understand that you're unhappy/angry/upset. I understand you've paid for a service you aren't receiving. I understand this causes you great stress. I understand that you want to make all of these things known. The unresolved question is whether you understand that you're using words incorrectly. It's not abandoned. It's not ignored. It just hasn't gotten fixed yet because other things are being done first. Is that fair? (shrug)
TYOC035 : Last Updated 09/13/2022 16:39
TYOC040 : Last Updated 09/21/2022 07:03
Actually.035 offline more earlier than that.Do you believe that without ignored?
You said that i m unhappy/angry/upset.But it is not my feeling because it is useless for the outage.In fact with more speechless and helpless.
This is a (7) day notice. The change will be completed on Thursday, September 29, 2022. Your service will be rebooted on that date for changes to take effect, although you may be able to perform this sooner by referring to the additional information provided below.
Current IP: 149.x
Future IP: 47.x
Additional Information:
A new IP address has been assigned to your virtual server as a secondary address, but it is not yet functional and should not yet be used. This address is provided early to allow you to plan ahead. Once it becomes functional, we will change your service's main IP address automatically, and it will appear as the main IP for your service and on your control panel. We will also update the network status page as they become available. When checking the network status page, your service node name (LAXA032) will be mentioned. At this time, both IP addresses will function during this transition period so you may utilize the "reconfigure networking" button early (optional.) Then, finally, on September 29th, we will automatically reconfigure your IP address.
Why is my IP address changing?
Our current IP provider made policy changes requiring a fee to be paid in cases of reported abuse. To avoid having to pass down these fees, and to ensure continued stability, we have leased addresses from another company for a longer term.
What if the new IP does not ping?
This means that the network configuration needs to be updated. We will post a guide on the network status page by the time the new IP address is available, so you can follow the guide in case of problems, but usually just clicking on the reconfigure network button on your service page works.
Will we receive IPv6?
We are also going to assign your service an IPv6 address, but this will be done at a later time. We will automatically assign these when ready and you'll see it on your service control panel when it is ready.
What if my service comes with additional IP addresses?
We will at a later time add a button where you can request back these additional addresses and pro-rated credits for the time they were unavailable. This button will appear on your service details page when it is ready.
If you have any questions, please let us know. Please use this specific ticket for any issues you have related to the IP change only and use another ticket for other existing issues instead. We apologize for the inconvenience.
Comments
Plot twist! IP Protection is turned on and @AlwaysSkint actually typoed his dns during the re-ip and he's actually hitting someone else's server. :-D
SolusVM bug in this case. There's a lot of different bugs. This instance happened to have more than one. Basically at any given time it's possible for it to skip one of the rules or assignments for whatever reason and it creates a chain reaction over time. There's another similar bug with disks that they never fixed, they just gave us a workaround which we haven't used as it's very I/O intensive and could kill the disk more quickly.
Some IP stealer also simultaneously took advantage of the situation, whether on purpose or not.
So there were essentially three people competing over one IP. How IP stealing usually can happen with the protection turned on is if one of the bugs above exists. Sometimes it also fully fails but that's more rare. It definitely didn't help that we also had to turn it off for some time when the NICs were burning up from the other issue we faced before splitting VLANs. And the short version of that goes: integrated NIC isn't amazing, so it struggles keeping up with so many ARP packets flying around, and once we turned on the protection it added another layer of things it had to keep track of which made it want to melt.
Fun stuff.
(edit) I suppose it's not necessarily a bug, it was just never coded to handle all the weird scenarios that may occur and doesn't clean up automatically. After all, SolusVM in its current state is someone used PHP where PHP probably shouldn't have been used and a bunch of low end hosts such as myself decided to use it anyway. Back in the 1800's when it was coded though maybe it was the only option.
Mentally strong people put each KVM in a separate VLAN.
Each IPv4 /32 or IPv6 /56 is individually routed to the VLAN.
Gateway address is a link-local address, for both IPv4 and IPv6.
IP stealing is impossible and no IP is wasted for gateway.
We accept Karma donations for the last flan. 🍮 affbrr
LOL. You know me too well!
Thanks for sorting it out Virmach. Now where's that big hammer?
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
TYOC040 Still updating the announcement, very good. TYOC035 have been offline for more than a month.
Status Offline
Node Name TYOC035
TYOC035 (Outage)Critical
Affecting Server - SolusVM
09/13/2022 16:39 Last Updated 09/13/2022 16:39
This issue affects a server that may impact your services
TYOC035 is facing a partial outage. We are evaluating a potential maintenance window.
Yes.TYOC035 has been abandoned
You know I've thought long and hard about it and decided you were right, I was deceiving myself. I want you to know I appreciate you pointing this out.
i think virmach has to fix the MJJ Tokyo unorthodox migration influx in parallel with getting the nodes back up to have a better control of the situation.
I bench YABS 24/7/365 unless it's a leap year.
Do u have any service in TYOC035?
The migration killed all maintenance progress.
No, and for that you have my sympathy. Truly unlucky for you.
So.This is why u can not understand.
I understand fine. I understand that you're unhappy/angry/upset. I understand you've paid for a service you aren't receiving. I understand this causes you great stress. I understand that you want to make all of these things known. The unresolved question is whether you understand that you're using words incorrectly. It's not abandoned. It's not ignored. It just hasn't gotten fixed yet because other things are being done first. Is that fair? (shrug)
Currently 22.5% of TYOC035 is potentially impacted.
Currently 40% of TYOC040 is potentially impacted.
That means 77.5% of TYOC035 isn't and 60% of TYOC040 isn't. Even with this, we've still been getting so many tickets about them when there's already a network status page posted and it can also be seen here on this thread with how often it's getting mentioned when in reality it's not impacting as many people as you'd assume when compared to other topics that have been discussed.
Right now SJCZ004 outage is impacting more people than both of those nodes combined. Tell me how many times that has been mentioned versus TYOC035 and TYOC040.
SEAZ009 also went down and it's impacting more people as well. How many mentions for that? We received infinite times more tickets about TYOC035 and TYOC040 (because it's 0, since people looked at the network status page for it.)
If we're already having a high support load as a result of just 35% of the people being affected, then by taking it offline to do further maintenance that's essentially guaranteed we'll receive 3x as many reports from 100% of the people being affected. Last time for TYOC040 we sent an email, made a network status, and even asked people to consider not contacting us so we can focus on the issue and we still got many requests. So while I understand it's clearly a big problem and I apologize for the continued outage for those people, understand that when we're being simultaneously bombarded with requests it's counterproductive and that we also have other issues we're tending to so it makes it a better choice to prioritize those at the moment when the only thing that's essentially guaranteed to come from another maintenance or migration attempt is a lot of tickets. Even if we migrate people off, we're going to end up getting a flood of tickets for that so we definitely do not want to continue having ticket backlogs and have to keep that in mind.
Sounds fair to me but then again I'm not an impacted party.
E-mails sent? I've received nothing, did I win a jackpot and non of my VPSes will change IP?
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
I've been in a bit of a predicament with VirMach for a while... For a start, I understand that VirMach has been having a lot of trouble lately with ColoCrossing - speaking of which, what happened to them? I have (or had) a dedicated server, which I haven't had access to for nearly 6 months because of a networking issue. Then last month, overnight, I got an email saying it was suddenly being "migrated" elsewhere, and that I should back up my data. Well this is the problem, I couldn't back it up, since I hadn't had access to it for months on end. Now I'm being told that the data is lost and there's little chance of it being recovered. So what happens now?
We accept Karma donations for the last flan. 🍮 affbrr
I was affected too. Follow the instructions in the email (also copied in this comment: https://lowendspirit.com/discussion/comment/94512/#Comment_94512, additional info here: https://lowendspirit.com/discussion/comment/94592/#Comment_94592) to create a ticket to (hopefully) get a new service deployed quickly. A ticket should have been generated for you automatically, but maybe it got missed/buried and put on the back burner with everything else going on, so best to open that priority support ticket. Best of luck!
Ohh and yeah -- getting the data back isn't going to be an option at this point.... ColonCrossing made sure of that.
yabs daddy
Disk is full on my Seafile instance.
The only machine with more disk is VirMach.
I spent all night moving the files.
Hopefully MIAZ011 doesn't lose data going forward.
Hourly backup is in place.
We accept Karma donations for the last flan. 🍮 affbrr
When will the sentencing of Tokyo's stowaways take place?
For dedicated servers too? Yikes.
Can someone tell me what MJJ is? Malaysian Jiu-Jitsu? I only know about the Brazilian kind.
If I understand the Tokyo stowaway thing, it sounds crazy that they haven't been booted, but it is also a serious software bug that it could happen at all.
I gotta add, both of my SJC vps are up now, have been up for the past week or two, and performing extremely well, as if the host nodes are nearly unloaded. Both have had IP changes and needed reinstalls but with that out of the way things are running smoothly. There may have been network issues that I haven't noticed since I've been idling both, after migrating a service away from one during the outage. I plan to bring that service back and see what happens.
@Virmach I know that you've said in the past that Transfers can be time consuming, but..
If you fancy a change from the "mundane" usual Tickets, could you please process the (merged) open Ticket for one.
Poor @vyas already paid me at the beginning of the month and it doesn't feel right holding onto his bucks, for nothing.
Go on, you know you want to.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@Virmach I was glad to see ipv6 working at Germany, thanks.
In other news, a Ryzen Migration from ATL to SEA went flawlessly.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@AlwaysSkint said:
West coast!?! Have you gone mad? ;-)
Aye! A wee experiment, as I have a client in that neck of the woods.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
did the same from LAX to AMS. now happy with xTom.
will Virmach plan to supply IPv6 in that location?
I bench YABS 24/7/365 unless it's a leap year.
I'm thinking that I could do with a second one there.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
its trademark MJJ , even more so for unorthodox migrants.
whats the issue with node 40? will you be extending service for those affected instead of credits? credits cant help me get back 6TB of premium bandwidth watching educational videos in Japan.
I bench YABS 24/7/365 unless it's a leap year.
its only 4ms away from my storage box with NAT IPv4, will be generating funny ideas when i catch a break 😏
I bench YABS 24/7/365 unless it's a leap year.
@VirMach sir, any info about adding ipv6 back to Tokyo?
I'm not sure if you're speaking of us or ColoCrossing in your comment but if it's us then the timeline doesn't correlate and it's possible you're facing some other layer of issues.
I believe we can share this information but I'll just say it based on what we had planned to potentially do rather than what we actually have done.
We wanted to give them a chance to consider powering these on, any servers that were paid for additionally through the end of August/September based on the vague communication we may have been provided regarding how they were going to proceed, whether or not it actually be what was supposed to happen. So that we could provide access to customers and reduce the damages caused. Since that did not happen you can make your own inferences on what happened.
For probably years at this point we've been trying to streamline the process. It obviously hasn't happened. It's just very easy for these tickets to stay indefinitely in limbo because everyone other than myself is essentially afraid to do it since it involves a lot of security. As in, all the red tape I put over it basically puts the #1 priority at making sure it's impossible for a hacked account to lose their service or for the wrong account to go to the wrong place or for a SolusVM account with multiple services to accidentally be transferred instead of just the one service and so on. It's a lot of triple checking everything, all the logs on each account, sometimes pages of it, and so on.
I know some people might think we just do it to charge $3 and then profit and then just do nothing for weeks or months but that's not really the case.
I didn't do anything, sorry, lol. I'm actually surprised it works, it shouldn't but I guess it probably does on one node still. May I know which node it is so we can try to preserve it?
IPv6 will be in all locations except potentially some delay in Hivelocity. Their IPv4 pricing is crazy so I'm not sure how it's going to be for IPv6 but I can't imagine it'd be too terrible. We already have large blocks with everyone else. QN, I think has the smallest IPv6 blocks assigned to us. So it goes xTom --> Dedipath --> QN --> Hivelocity in terms of likelihood for problems with direct providers based on the quantity we have per cabinet I think right now.
The plan is to do 1x IPV6 and 1x /64 IPv6 subnet. Technically with some partners we have enough to where we could assign more than /64 IPv6 such as a /56 IPv6 but I think that's a good amount (/64 IPv6.) Open to suggestions.
We also want to set up a failover IPv4 NAT for every VPS as well. I'm trying to see if we can essentially purchase one IPv4 block for this so it's actually ours, owned by us, and do that with it so no matter what customers will never lose full connectivity just as a result of some third party deciding to pull IPv4. And then that essentially will also open the door to ultra low priced NAT offerings with IPv6. Think $7/YR instead of $12/YR with IPv4.
The issue will require several reboots and potentially hard power off and reseating. One of the disks fully disappeared and for some reason the system isn't throwing errors which makes... no sense really but I think I've seen it happen maybe once before a long time ago. Usually if a disk goes away in the OS it would throw errors saying it ever existed in the past and marked as [UNKNOWN] but this time it's not there and it's acting as if it was never there.
Sorry if that information sounds useless I just won't know more until we power it off to go into BIOS and check a few other things.
Credits, I don't know how we'll end up doing that. I don't want to get in another situation where we delay fixing it specifically to be able to go through manually and do any specific type of crediting because it becomes more difficult to track and credit after it's fixed especially if they move around.
IPv4 Update - We signed the contract with the new provider and working to get it done within our timeline. I know we're behind by a day so we'll push the notice into the change officially happening on Thursday instead of next Wednesday.
We've tested the first step and it looks to be going very smoothly. So you should get a ticket tonight with your exact IPv4 to expect.
i understand from your point of view, but as a customer its not that fair either. the only consolation is that the service is lowendpriced.
anyway i dont think there's any effective solution to this apart from getting the nodes up and stable as a priority.
in any case hope your other battles are going strong.
I bench YABS 24/7/365 unless it's a leap year.
We'll definitely get it figured out in terms of credit. I'm not saying we won't provide any, just that it may be done later so we can focus on the outage.
i just needed wanted 1xIPv6 so anything goes i guess?
although with /64 its a ready backup in case of any unforeseen IPv4 mishaps that may cripple services especially on a scale such as yours.
but NAT priced offerings? freaking awesome. it could be a win-win for true lowenders without provider bearing the liabilities of IPv4.
on that note i would suggest an "NAT port control panel" that can forward ports on the portal, instead of manually configuring it within the VPS. beginners like me wont have to sweat it then. i think this would fly.
having more control over IPs and Hardware, moving in positive direction amidst the teething issues.
I bench YABS 24/7/365 unless it's a leap year.
Phoenix has been notified of IP address change. The emails have been sent from SolusVM. This location is not yet getting the new IP address. We're preparing to send in the tickets (emails will come from WHMCS) for the other locations. These will have the new IP address included with them.
The plan was to originally provide 1 x /64 IPv6 but we decided to also provide a single IPv6 which both facilitates using it through SolusVM and is more user friendly. Adding in the 1x IPv6 is easier than adding the /64 IPv6 so a long time ago I believe I said that the /64 IPv6 may not be guaranteed but currently I don't see any reason why this wouldn't be possible.
Yeah we've been meaning to do this for quite some time but in the past with no IPv6 and the fact that SolusVM doesn't natively support it, we ended up not doing it. But once we finally get past all this craziness it shouldn't be too much of a headache to set up a script that configures it automatically and with IPv6 it's a completed solution.
Yeah, I'm very excited about this part of our company's lifetime.
My Gmail received the email (just reporting)
Is it all of Phoenix that is changing? Haven't gotten the email yet for one on PHXZ002
If you have an ASN, getting your own v6 space should be pretty easy still. It is v4 that is difficult.
This doesn't sound worthwhile. v6-only is almost as good, in the event that the VPS's dedicated v4 address goes away. Whatever abuse was going on will likely take out the shared v4 address too. That happens all the time with NAT services. LES got its start that way and super cheap NAT VPS is fun, but was never very reliable because of the shared address being an abuse and DDOS target.
Just got email about change for a MIAZ012 vps as a note
Edit: others rolling in as well for NY
TYOC035 : Last Updated 09/13/2022 16:39
TYOC040 : Last Updated 09/21/2022 07:03
Actually.035 offline more earlier than that.Do you believe that without ignored?
You said that i m unhappy/angry/upset.But it is not my feeling because it is useless for the outage.In fact with more speechless and helpless.
Hello sexy,
This is a (7) day notice. The change will be completed on Thursday, September 29, 2022. Your service will be rebooted on that date for changes to take effect, although you may be able to perform this sooner by referring to the additional information provided below.
Current IP: 149.x
Future IP: 47.x
Additional Information:
A new IP address has been assigned to your virtual server as a secondary address, but it is not yet functional and should not yet be used. This address is provided early to allow you to plan ahead. Once it becomes functional, we will change your service's main IP address automatically, and it will appear as the main IP for your service and on your control panel. We will also update the network status page as they become available. When checking the network status page, your service node name (LAXA032) will be mentioned. At this time, both IP addresses will function during this transition period so you may utilize the "reconfigure networking" button early (optional.) Then, finally, on September 29th, we will automatically reconfigure your IP address.
We will post any additional updates on the network status page: https://billing.virmach.com/serverstatus.php
FAQ:
Why is my IP address changing?
Our current IP provider made policy changes requiring a fee to be paid in cases of reported abuse. To avoid having to pass down these fees, and to ensure continued stability, we have leased addresses from another company for a longer term.
What if the new IP does not ping?
This means that the network configuration needs to be updated. We will post a guide on the network status page by the time the new IP address is available, so you can follow the guide in case of problems, but usually just clicking on the reconfigure network button on your service page works.
Will we receive IPv6?
We are also going to assign your service an IPv6 address, but this will be done at a later time. We will automatically assign these when ready and you'll see it on your service control panel when it is ready.
What if my service comes with additional IP addresses?
We will at a later time add a button where you can request back these additional addresses and pro-rated credits for the time they were unavailable. This button will appear on your service details page when it is ready.
If you have any questions, please let us know. Please use this specific ticket for any issues you have related to the IP change only and use another ticket for other existing issues instead. We apologize for the inconvenience.
Powerful AMD Ryzen VPS (aff)