@VirMach said:
Okay so I've worked on TYOC040 probably way too long and it just hit me. I don't know if you guys remember but I said something weird is going on with the disks because a bunch of LVMs are missing and it was fixed and has been working/appears as it's all fine. Then I was thinking about TYOC002S and how that's also acting a lot different in terms of usage patterns for the region and finally thought "wait, there's no way up to 40% of the people are offline with missing LVMs because ...? No, but then for how many tickets we've been getting about it enough people would have mentioned they did that since it's a pretty important thing to mention, right?" Well I don't know, maybe some people did but I'd say we got pretty much every type of ticket possible for this, from not pinging, GFW concerns, VPS is offline, doesn't work, it's been down for 1 day, it's been down for 2 days and so on, outage report, and so on, over and over.
And then I actually went through and checked every single one.
Yep, turns out it is. It looks like what happened was at some point in time, everything was normal/fixed. Then, a disk dropped off (it timed out.) So online VMs would have seen I/O errors and usually what would happen is people would report that, or make a ticket saying they have important data, etc.
But this is Tokyo and the use case is different. What would cause 40% of the LVMs to just disappear? Make your guess now and remember, this is logs from just about 40% of virtual servers on a single node.
24/09/2022 03:43 AM Reinstall OS
01/09/2022 2:32 AM Reinstall OS
28/08/2022 09:21 PM Boot
28/08/2022 09:16 PM Reinstall OS
20/08/2022 04:48 AM KVM reconfigure
20/08/2022 04:45 AM Boot
20/08/2022 04:45 AM Change Boot Order
20/08/2022 04:45 AM Change Boot Order
20/08/2022 04:34 AM Reinstall OS
20/08/2022 04:33 AM KVM reconfigure
15/08/2022 06:21 AM Reinstall OS
15/08/2022 12:54 AM Reinstall OS
14/08/2022 06:33 PM Boot
14/08/2022 06:06 PM Reinstall OS
14/08/2022 10:20 AM Reboot
@VirMach said:
Honestly I'm just waiting for someone to quote it and ask an unrelated question.
Let me be first (without quoting)
It's long monthly log and I know it's just 40% as you said, but seriously, does five reinstalls, four reboots and some unmounting or changing boot order (not by a single client, but all together) in a whole day kill server performance? :P
Ok, some days are more busy, some less, but that's expected as those servers are provisioned for most clients in the same period.
TBH, If I'm trying something out, I've seen me doing multi-installs within an hour. Sometimes I need to boot into rescue to fix stuff too. I can envisage a problem if folks are doing the same thing, all at the same time, otherwise...
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said:
Honestly I'm just waiting for someone to quote it and ask an unrelated question.
Let me be first (without quoting)
It's long monthly log and I know it's just 40% as you said, but seriously, does five reinstalls, four reboots and some unmounting or changing boot order (not by a single client, but all together) in a whole day kill server performance? :P
Ok, some days are more busy, some less, but that's expected as those servers are provisioned for most clients at the same day.
A re-install while the disk is away causes the LVM to get deleted and not re-created per SolusVM coding. Even if the disk is functional there are times where several commands in a row cause it to glitch out completely.
In other regions, this would usually not be a problem because the first solution to I/O errors usually isn't spamming an average of like 30 buttons. I'm just surprised 40% of the people on the node actually did that, especially since this was one of the first nodes on Japan and we already had to go through one round of recreations as a result. And I'm surprised I didn't catch on quicker. Maybe it was on the back of my mind but I guess I refused to believe 40% of the people on the node would participate in the frantic button clicking and re-installs as the default solution.
I mean at the same time it can be a good thing, it means if the disk was actually lost, almost no one would care since it's probably just running a VPN and not important data. I just wouldn't have guessed 40% of the people on the node would have done it. And it's not even that 40% did it, it's more like 95% of the people that were on the disk that dropped off and then of course a portion of that were from Ryzen migrates back and forth. We definitely need to stop it from being a possibility though.
@VirMach said: A re-install while the disk is away causes the LVM to get deleted and not re-created per SolusVM coding.
Ahh, the salient part.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said:
Okay, TYOC040 LVMs have been regenerated. Feel free to resume spam re-installs.
no permanent solution?
Since the original maintenance (not the original original but the most recent one before this round) it's already been stable again. And anyone who didn't re-install would have returned online since then. There's a few other cases where it could have happened as well but I'd say almost everyone who was offline was basically offline this whole time because they re-installed sometime around August 12th and 22nd maybe. Could be some other time, but around there.
I guess only re-install when necessary and not as a solution to your problems, if you know you previously had a functional OS, especially don't do it to fix I/O issues, and we'll probably work on preventing people from doing it if the node is facing potential disk anomalies (until they edit the disabled button and do it anyway.)
As for on SolusVM directly I don't think there's much we can do.
@VirMach said: ..until they edit the disabled button and do it anyway.
You need a honeypot, you do. Instaban for those who dip their paw in.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@foitin said: I'm shocked that you allow copyright stuffs from Japan hosted on domestic severs. real bold move
Even doujinshi (fanmade) adult manga/douga is the object of investigation (CP regulation and copyright infringements)
Someone reports to the police and your upstream will cut your connection even your contract.
one keyword for you: manga-mura (漫画村) which I believe is not even 100% hentai
no,h@h only works for e-hentai,it handles DMCA and cp.
edit:to load pictures from a h@h server,you need to open e-hentai,then e-hentai assign a h@h node that has this picture,and you get the url,url is long,and the hostname will change(maybe after reboot),so i don't think it is a "public" service
so i don't think it is a "public" service
edit2:so owner will just send DMCA to ehentai,not to a random h@h server
@VirMach said: ait so does it work with a big disk and then a smaller disk as cache? Because that would make sense and how people should be setting these up. It's like if
yes,i think it will has some "trending" file,maybe we can put it into cache(even ramdisk...)
can anyone recommend a cache software that can mount(like rclone)
and,i am storing all my H@H files in my google drive,but team drive only allow to save 500000 files.anyone know what software can put bunch small files into a block?(for example,many files put into a 4MB block file)
My DALZ005 changed ip in control panel but as I installed ubuntu 20.04 server from iso, it looks like IP refused to change nor primaty not 2nd one work, reconfigure network did not help
I tried VNC but for some reason turboVNC shot corrupted screen (you can see ubuntu boot but then it get corrupted) i need too see how to fix that first
I never have an issue with HTML VNC in the SolusVM control panel. Occasionally I use SSVNC/Remmina (Linux, of course) - primarily for copy/paste and screen resolution reasons.
I mostly use ISO installs and manual network configuration, so don't have the expectation that SolusVM will correctly reconfigure the IP addresses.
The following locations are still awaiting the datacenter to either announce the new IP addresses or to fix a network configuration.
LAX2, NYC, SEA, DAL, ATL, SJC
So, you may/may not get lucky. Do you feel lucky; well, do you?
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Yeah I seen it, maybe network IPs are not yet "online"
VNC worked fine for me for system reinstall but looks like the ubuntu 20.04 server I installed have some issue with VNC, I tried few clients all show black screen and color garbage in top of screen
I'm facing issues both on TYOC002S and TYOC026.
On TYOC002S, my server was suspended on 9/26. However, I don't think I could use up 20TB traffic. The statistics on SolusVM also shows that I didn't use much traffic after the initial setup process.
On TYOC026, suddenly ssh requires password for me, and I tried to reboot in SolusVM. Then the boot disk has gone. I'm not sure if TYOC026 is in maintenance now.
@reb0rn said: Yeah I seen saw it, maybe network IPs are not yet "online"
Just tried pinging the new gateway on DALZ007 - not active yet, it seems.
@reb0rn said: ubuntu 20.04 server I installed have has some issue with VNC
I have some issues with Ubuntu as a server OS - Debian, thanks.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach said: if you know you previously had a functional OS, especially don't do it to fix I/O issues, and we'll probably work on preventing people from doing it if the node is facing potential disk anomalies (until they edit the disabled button and do it anyway.)
That is like counter-intuitive to me. I know I had working system, but one of the disk failed, you fixed that (networkstatus), I don't really need old files - I will just reinstall. I would really expect that this mythical LVM you talking about that and I have no idea what it is (I just press buttons in panel, duh) was fixed along the disk and reinstall would just work.
Especially when support is overwhelmed and take weeks to answer - reinstalling system seems like 'safe & correct' way to go before engaging Support - it should place me on other disk or on this fixed one or somethingsomething. No one knows SolusVM is 10+ years old piece of crap that takes years to make sense.
You can't really be mad a people for trying to reinstall. You can be mad for people spamming reinstall every minute ;')
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
@VirMach said: if you know you previously had a functional OS, especially don't do it to fix I/O issues, and we'll probably work on preventing people from doing it if the node is facing potential disk anomalies (until they edit the disabled button and do it anyway.)
That is like counter-intuitive to me. I know I had working system, but one of the disk failed, you fixed that (networkstatus), I don't really need old files - I will just reinstall. I would really expect that this mythical LVM you talking about that and I have no idea what it is (I just press buttons in panel, duh) was fixed along the disk and reinstall would just work.
Especially when support is overwhelmed and take weeks to answer - reinstalling system seems like 'safe & correct' way to go before engaging Support - it should place me on other disk or on this fixed one or somethingsomething. No one knows SolusVM is 10+ years old piece of crap that takes years to make sense.
You can't really be mad a people for trying to reinstall. You can be mad for people spamming reinstall every minute ;')
@VirMach said: if you know you previously had a functional OS, especially don't do it to fix I/O issues, and we'll probably work on preventing people from doing it if the node is facing potential disk anomalies (until they edit the disabled button and do it anyway.)
That is like counter-intuitive to me. I know I had working system, but one of the disk failed, you fixed that (networkstatus), I don't really need old files - I will just reinstall. I would really expect that this mythical LVM you talking about that and I have no idea what it is (I just press buttons in panel, duh) was fixed along the disk and reinstall would just work.
Especially when support is overwhelmed and take weeks to answer - reinstalling system seems like 'safe & correct' way to go before engaging Support - it should place me on other disk or on this fixed one or somethingsomething. No one knows SolusVM is 10+ years old piece of crap that takes years to make sense.
You can't really be mad a people for trying to reinstall. You can be mad for people spamming reinstall every minute ;')
Yes but I can't think of any non-rare situation where an OS re-install would help at all in any situation where the OS/VM was previously functional and abruptly stopped functioning to that level, without anything having been done first. Let me know if you can think of any cases where it would.
There's a difference between doing it after something did occur or was announced to occur.
I understand there's a point in time where network status was not made either, but a portion of these also occurred after it was announced. If there's ever an issue with a server it's generally a big no-no to use any functions. I'm not blaming anyone though, I get it. It's just something we need to watch out more for in Tokyo as apparently there will be a high rate of that going on. And of course people can and will do it and it'll just extend their downtime every time no matter what because logically, it will always have to be fixed first before we move on to the next step of regenerating anyone who regressed it into having no LVM.
Again, I know it may have been helpful after migrations or after a specific OS didn't first install, but that was an event that occurred. And of course a small portion will be those that just needed an OS re-install anyway.
(edit) So yeah, I definitely don't think it's counter-intuitive but I also don't expect people to follow the suggestion. That is the correct suggestion though, kind of like our previous suggestion to not use the Ryzen migrate button if the node is reported as having issues.
@lxm said: @VirMach AZ004 network become terrible ,package loss
CHI2 as well, should improve once the swaps are complete. These two locations still need switch configuration changes so while they have additional blocks there will be additional packets until the old IPs are no longer used.
Also additional issue, but expected: routing globally will take some hours to a day or so to improve.
Bleh, must wake up before diagnosing issues. Network interface was ens3 instead of eth0 so auto fix didn't work and then I made a couple typos fixing it all for a NYC vps
@VirMach Just a note on ATLZ007B.
I can ping the new gateway with the new IP, but when I try to go anywhere else I get:
"From 128.136.83.193 icmp_seq=X Packet filtered"
Almost every time I install debian 10-11 in those migrated VPS, it works fine with initial OS installation but immediately lost the disk with full OS upgrade and reboot (yes, only apt-get upgrade), Only can discover this patten after many failures of Debian setup on those migrated nodes, in the end all most all my VPS are reinstalled with Ubuntu which have no issue of disk lost after the upgrade and reboot, it was the case at least few weeks ago maybe no longer the case for now, for reference.
@VirMach said: Yes but I can't think of any non-rare situation where an OS re-install would help at all in any situation where the OS/VM was previously functional and abruptly stopped functioning to that level, without anything having been done first. Let me know if you can think of any cases where it would.
It appears that LAXA031 has had its IPs reconfigured, but I still cannot access the control panel for my VM (90s timeout). I manually reconfigured the IP and set it as main a few days before this mass migration. Is this still in limbo or should I create a ticket?
We were able to get some people off but it went offline pretty quickly. I've already requested the DC try to bring it back up and monitoring so we can continue the emergency migrations.
@VirMach
What is wrong with SJCZ005?
It's been a long time, what is your plan for SJCZ005?
We were able to get some people off but it went offline pretty quickly. I've already requested the DC try to bring it back up and monitoring so we can continue the emergency migrations.
@VirMach
What is wrong with SJCZ005?
It's been a long time, what is your plan for SJCZ005?
I was actually about to post an update on it. And yes planned on moving people off a few days ago, doing that now, and we'll be crediting people for a month or extending service by a month.
Also wanted to update everyone else:
Some of these already sent out as emails, network updates, etc.
Old LAX servers that failed: All have been regenerated on Ryzen. We'll be crediting people 6 months or extending by 6 months. They still need IP addresses assigned, will most likely get that part done by Monday, couldn't get to it today.
LAXA031: Maintenance scheduled with DC. We thought it stabilized after maintenance but there was miscommunication there, it just happened to stabilize and then ran into issues again last week or so, we've scheduled they perform original intended hardware maintenance and that should hopefully stabilize it. Crediting people 1 month or extending.
SEAZ009 - Scheduled maintenance, it may fix the issue. Extending/crediting 1 month.
SEAZ010 - This one has been extended because we're having trouble getting DC hands to perform what we requested. It's "only" been about a week now but I'm completely blind on this one at the moment, we asked them to just ship it out to us so we can get the data off and people migrated sooner if they can't perform our request (I asked them to check what's being displayed last and got back "this has been completed.") No auto credit/extension if we can get this one fixed by Monday/Tuesday but you can of course request SLA credits in a ticket.
SJCZ005 - Also covered above. Trying to migrate people now. If that doesn't work I have a final idea/attempt at getting it fixed permanently as well. Migrations will be to LAX as we have limited space in SJC. Also 1 month credit/extension here.
@FrankZ said: @VirMach Just a note on ATLZ007B.
I can ping the new gateway with the new IP, but when I try to go anywhere else I get:
"From 128.136.83.193 icmp_seq=X Packet filtered"
128.136.83.193 is a Flexential IP
So nothing in or out beyond the gateway.
The 8 other IP changes went just fine.
Yeah, network's being weird but ATLZ003 was also completely going wacko in multiple ways so I had to focus most the time on that and everything else being completed today. I am seeing bursts out but I've also verified the issue you describe several hours earlier, didn't have any time to update especially since I'm trying to figure out the timeline. It does look like it was announced so it must be misconfiguration on the network equipment. I swear I cleared it though so it must have been working and they also checked on their end, we'll see.
I have VPS on SJC2005 (down) and SJC2008 (older servier, working again after being down for months, but idling and expiring soon since I don't need two VPS at that location). I'd rather not get migrated to LAX if there is some hope of SJC2008 becoming reliable.
1) Instead of migrating the SJC2005 vps to LAX could it be transferred to SJC2008, cancelling the existing SJC2008 vps, leaving just one?
2) Rather than service extensions could it be possible to have a resource increase for that remaining vps, from 768MB to 1.5GB?
If either of these vps gets migrated or transferred, it's not necessary to actually transfer the data, if it's easier at your end to just delete the old vps and create a new one that I can reinstall on. Thanks.
Yeah, network's being weird but ATLZ003 was also completely going wacko in multiple ways so I had to focus most the time on that and everything else being completed today. I am seeing bursts out but I've also verified the issue you describe several hours earlier, didn't have any time to update especially since I'm trying to figure out the timeline. It does look like it was announced so it must be misconfiguration on the network equipment. I swear I cleared it though so it must have been working and they also checked on their end, we'll see.
Ah the "remote hands with no brains attached" thing again. I fled Dallas but my Atlanta one is working well usually. And I don't need more in NY lol
This has been an interesting adventure to see just how badly some DC and remote hands setup can be. Pretty sure Virmach would describe it in slightly less interesting and more rage inducing terms though.
@Daevien said:
This has been an interesting adventure to see just how badly some DC and remote hands setup can be. Pretty sure Virmach would describe it in slightly less interesting and more rage inducing terms though.
Most remote hands techs are idiots across the board. If you need their help often or for something, most of the time it's cheaper to fly to the DC yourself or just to hire contractors. I can think of maybe 10 experiences I've had with datacenter hands that I'd consider positive, out of hundreds of interactions.
Followup to above: I spoke too soon, Comcast blocks the new Buffalo IP too. It's annoying. It is one of the old 384MB specials and I have a small personal website on it. I guess I can try to migrate it to another location? I'm not sure if that is supposed to work.
Does anyone know the current state of VPSShared? That's where my site was before, but it stopped working for a while due to the shared IP being null routed. I saw in the control panel that the IP changed after a while, but the new one doesn't work either.
@fluttershy said:
Most remote hands techs are idiots across the board. If you need their help often or for something, most of the time it's cheaper to fly to the DC yourself or just to hire contractors. I can think of maybe 10 experiences I've had with datacenter hands that I'd consider positive, out of hundreds of interactions.
True I guess, I haven't used them many times personally, usually just depending on either my own ipkvm or getting one connected temp.
When I think about it, I've probably been the remote hands to help out someone more times than I've used them myself
@willie said:
Followup to above: I spoke too soon, Comcast blocks the new Buffalo IP too. It's annoying. It is one of the old 384MB specials and I have a small personal website on it. I guess I can try to migrate it to another location? I'm not sure if that is supposed to work.
Comcrap just flat out blocks the ip entirely? Thats pretty bad even for them.
Does anyone know the current state of VPSShared? That's where my site was before, but it stopped working for a while due to the shared IP being null routed. I saw in the control panel that the IP changed after a while, but the new one doesn't work either.
Last I remember was an update about the changeover in IP and I thought it was working again, I don't use them myself.
@willie said:
I have VPS on SJC2005 (down) and SJC2008 (older servier, working again after being down for months, but idling and expiring soon since I don't need two VPS at that location). I'd rather not get migrated to LAX if there is some hope of SJC2008 becoming reliable.
1) Instead of migrating the SJC2005 vps to LAX could it be transferred to SJC2008, cancelling the existing SJC2008 vps, leaving just one?
2) Rather than service extensions could it be possible to have a resource increase for that remaining vps, from 768MB to 1.5GB?
If either of these vps gets migrated or transferred, it's not necessary to actually transfer the data, if it's easier at your end to just delete the old vps and create a new one that I can reinstall on. Thanks.
SJC is full due to SJCZ004 also having more serious issues, so we used all the space there to SJC. It hasn't exactly been smooth sailing for that location so there's only so much room we have left to move around within SJC.
SJCZ005 is technically more salvageable at the moment. It has to do with a loose or malfunctioning cable. It was hard to diagnose for a while because of the sporadic manner it manifested itself but I finally was able to catch it. We did have maintenance done on this node before so perhaps when they opened it up to fix the previous issues, a cable shifted and it worked fine (until it didn't.) Resource increase would be too complicated to do in bulk but I'll do it for you specifically (@willie) since you've been stuck with us in the worst string of luck ever.
Yeah, network's being weird but ATLZ003 was also completely going wacko in multiple ways so I had to focus most the time on that and everything else being completed today. I am seeing bursts out but I've also verified the issue you describe several hours earlier, didn't have any time to update especially since I'm trying to figure out the timeline. It does look like it was announced so it must be misconfiguration on the network equipment. I swear I cleared it though so it must have been working and they also checked on their end, we'll see.
Ah the "remote hands with no brains attached" thing again. I fled Dallas but my Atlanta one is working well usually. And I don't need more in NY lol
This has been an interesting adventure to see just how badly some DC and remote hands setup can be. Pretty sure Virmach would describe it in slightly less interesting and more rage inducing terms though.
Yeah I had a lot more written but realized I was going off on a weird upset tangent so I cut it out. There's a reason I'm unsurprised that such a thing would happen in Atlanta, let's just leave it there.
It runs as well as my new 10gb nic does. Oh sorry, my new 10gb paperweight. sigh Here's hoping the others incoming work better lol
LAXA014 was so close to being done around Thursday/Friday. Then it crapped out mid-migration and SolusVM migration feature fully malfunctioned as a result as well, making it more difficult to do because we were doing it in batches versus a bulk script since we knew it's unstable. So the first batch was like 70% done, then it crashed, and made the system lock up. I fixed it in every way we knew possible but now we're facing another weird bug with the migration tool, so it just means we have to plan out semi-bulk manual batches instead which complicated it enough to where with everything else going on, it definitely didn't get done by Friday.
@willie said:
Followup to above: I spoke too soon, Comcast blocks the new Buffalo IP too. It's annoying. It is one of the old 384MB specials and I have a small personal website on it. I guess I can try to migrate it to another location? I'm not sure if that is supposed to work.
Does anyone know the current state of VPSShared? That's where my site was before, but it stopped working for a while due to the shared IP being null routed. I saw in the control panel that the IP changed after a while, but the new one doesn't work either.
The null thing... oh god, I don't want to even get into it but I'll just say one thing: we tried. Many times.
New one shouldn't get nullrouted like that. The IP change for those also went through and there should not be any immediate problems for a long time. We just are still setting up rDNS for mailing but other than that, everything looks good and if it doesn't then it'll be an easy fix as it's not facing any weird hardware issues or anything else as of right now.
@willie said: Followup to above: I spoke too soon, Comcast blocks the new Buffalo IP too. It's annoying. It is one of the old 384MB specials and I have a small personal website on it. I guess I can try to migrate it to another location? I'm not sure if that is supposed to work.
I have no idea what's going on with Comcast recently but I've had a ton of issues with them and routing from LAX office to pretty much everywhere. What's the IP start with?
Old LAX nodes that had catastrophic failure are getting their new IPs now. You can manually change the main IP to get it working sooner. I may be able to get the script working to do the main IP swaps for them but if something goes wrong that'll get done tomorrow.
(And the credits/extensions will be done tomorrow, it requires a lot of DB queries that I don't want to try to remember right now while I'm doing a lot of other things, sorry.)
Comments
...
Do u mean the TYOC035 is the same as TYOC040?
Let me be first (without quoting)
It's long monthly log and I know it's just 40% as you said, but seriously, does five reinstalls, four reboots and some unmounting or changing boot order (not by a single client, but all together) in a whole day kill server performance? :P
Ok, some days are more busy, some less, but that's expected as those servers are provisioned for most clients in the same period.
Does anyone else remember when browsers didn't have a minimum size for the scrollbar autoscale or am I implanting false memories in my head?
@Virmach gimme a break! Go drink more coffee.
TBH, If I'm trying something out, I've seen me doing multi-installs within an hour. Sometimes I need to boot into rescue to fix stuff too. I can envisage a problem if folks are doing the same thing, all at the same time, otherwise...
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
A re-install while the disk is away causes the LVM to get deleted and not re-created per SolusVM coding. Even if the disk is functional there are times where several commands in a row cause it to glitch out completely.
In other regions, this would usually not be a problem because the first solution to I/O errors usually isn't spamming an average of like 30 buttons. I'm just surprised 40% of the people on the node actually did that, especially since this was one of the first nodes on Japan and we already had to go through one round of recreations as a result. And I'm surprised I didn't catch on quicker. Maybe it was on the back of my mind but I guess I refused to believe 40% of the people on the node would participate in the frantic button clicking and re-installs as the default solution.
I mean at the same time it can be a good thing, it means if the disk was actually lost, almost no one would care since it's probably just running a VPN and not important data. I just wouldn't have guessed 40% of the people on the node would have done it. And it's not even that 40% did it, it's more like 95% of the people that were on the disk that dropped off and then of course a portion of that were from Ryzen migrates back and forth. We definitely need to stop it from being a possibility though.
Okay, TYOC040 LVMs have been regenerated. Feel free to resume spam re-installs.
Ahh, the salient part.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
no permanent solution?
I bench YABS 24/7/365 unless it's a leap year.
Since the original maintenance (not the original original but the most recent one before this round) it's already been stable again. And anyone who didn't re-install would have returned online since then. There's a few other cases where it could have happened as well but I'd say almost everyone who was offline was basically offline this whole time because they re-installed sometime around August 12th and 22nd maybe. Could be some other time, but around there.
I guess only re-install when necessary and not as a solution to your problems, if you know you previously had a functional OS, especially don't do it to fix I/O issues, and we'll probably work on preventing people from doing it if the node is facing potential disk anomalies (until they edit the disabled button and do it anyway.)
As for on SolusVM directly I don't think there's much we can do.
You need a honeypot, you do. Instaban for those who dip their paw in.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach
Really man? Stop posting large log.
https://microlxc.net/
Even doujinshi (fanmade) adult manga/douga is the object of investigation (CP regulation and copyright infringements)
Someone reports to the police and your upstream will cut your connection even your contract.
one keyword for you: manga-mura (漫画村) which I believe is not even 100% hentai
no,h@h only works for e-hentai,it handles DMCA and cp.
edit:to load pictures from a h@h server,you need to open e-hentai,then e-hentai assign a h@h node that has this picture,and you get the url,url is long,and the hostname will change(maybe after reboot),so i don't think it is a "public" service
edit2:so owner will just send DMCA to ehentai,not to a random h@h server
yes,i think it will has some "trending" file,maybe we can put it into cache(even ramdisk...)
can anyone recommend a cache software that can mount(like rclone)
and,i am storing all my H@H files in my google drive,but team drive only allow to save 500000 files.anyone know what software can put bunch small files into a block?(for example,many files put into a 4MB block file)
My DALZ005 changed ip in control panel but as I installed ubuntu 20.04 server from iso, it looks like IP refused to change nor primaty not 2nd one work, reconfigure network did not help
I tried VNC but for some reason turboVNC shot corrupted screen (you can see ubuntu boot but then it get corrupted) i need too see how to fix that first
I never have an issue with HTML VNC in the SolusVM control panel. Occasionally I use SSVNC/Remmina (Linux, of course) - primarily for copy/paste and screen resolution reasons.
I mostly use ISO installs and manual network configuration, so don't have the expectation that SolusVM will correctly reconfigure the IP addresses.
RTFM: https://billing.virmach.com/serverstatus.php
So, you may/may not get lucky. Do you feel lucky; well, do you?
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Yeah I seen it, maybe network IPs are not yet "online"
VNC worked fine for me for system reinstall but looks like the ubuntu 20.04 server I installed have some issue with VNC, I tried few clients all show black screen and color garbage in top of screen
I'm facing issues both on TYOC002S and TYOC026.
On TYOC002S, my server was suspended on 9/26. However, I don't think I could use up 20TB traffic. The statistics on SolusVM also shows that I didn't use much traffic after the initial setup process.
On TYOC026, suddenly ssh requires password for me, and I tried to reboot in SolusVM. Then the boot disk has gone. I'm not sure if TYOC026 is in maintenance now.
Just tried pinging the new gateway on DALZ007 - not active yet, it seems.
I have some issues with Ubuntu as a server OS - Debian, thanks.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Time for 035
That is like counter-intuitive to me. I know I had working system, but one of the disk failed, you fixed that (networkstatus), I don't really need old files - I will just reinstall. I would really expect that this mythical
LVMyou talking about that and I have no idea what it is (I just press buttons in panel, duh) was fixed along the disk and reinstall would just work.Especially when support is overwhelmed and take weeks to answer - reinstalling system seems like 'safe & correct' way to go before engaging Support - it should place me on other disk or on this fixed one or somethingsomething. No one knows SolusVM is 10+ years old piece of crap that takes years to make sense.
You can't really be mad a people for trying to reinstall. You can be mad for people spamming reinstall every minute ;')
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
this.
I bench YABS 24/7/365 unless it's a leap year.
Yes but I can't think of any non-rare situation where an OS re-install would help at all in any situation where the OS/VM was previously functional and abruptly stopped functioning to that level, without anything having been done first. Let me know if you can think of any cases where it would.
There's a difference between doing it after something did occur or was announced to occur.
I understand there's a point in time where network status was not made either, but a portion of these also occurred after it was announced. If there's ever an issue with a server it's generally a big no-no to use any functions. I'm not blaming anyone though, I get it. It's just something we need to watch out more for in Tokyo as apparently there will be a high rate of that going on. And of course people can and will do it and it'll just extend their downtime every time no matter what because logically, it will always have to be fixed first before we move on to the next step of regenerating anyone who regressed it into having no LVM.
Again, I know it may have been helpful after migrations or after a specific OS didn't first install, but that was an event that occurred. And of course a small portion will be those that just needed an OS re-install anyway.
(edit) So yeah, I definitely don't think it's counter-intuitive but I also don't expect people to follow the suggestion. That is the correct suggestion though, kind of like our previous suggestion to not use the Ryzen migrate button if the node is reported as having issues.
NYC is working now, updating network status after confirming all nodes.
@VirMach AZ004 network become terrible ,package loss
CHI2 as well, should improve once the swaps are complete. These two locations still need switch configuration changes so while they have additional blocks there will be additional packets until the old IPs are no longer used.
Also additional issue, but expected: routing globally will take some hours to a day or so to improve.
Bleh, must wake up before diagnosing issues. Network interface was ens3 instead of eth0 so auto fix didn't work and then I made a couple typos fixing it all for a NYC vps
@VirMach Just a note on ATLZ007B.
I can ping the new gateway with the new IP, but when I try to go anywhere else I get:
"From 128.136.83.193 icmp_seq=X Packet filtered"
128.136.83.193 is a Flexential IP
So nothing in or out beyond the gateway.
The 8 other IP changes went just fine.
so far all ip changes went smoothly on all affected servers
My experience after ryzen migration.
Almost every time I install debian 10-11 in those migrated VPS, it works fine with initial OS installation but immediately lost the disk with full OS upgrade and reboot (yes, only apt-get upgrade), Only can discover this patten after many failures of Debian setup on those migrated nodes, in the end all most all my VPS are reinstalled with Ubuntu which have no issue of disk lost after the upgrade and reboot, it was the case at least few weeks ago maybe no longer the case for now, for reference.
It appears that LAXA031 has had its IPs reconfigured, but I still cannot access the control panel for my VM (90s timeout). I manually reconfigured the IP and set it as main a few days before this mass migration. Is this still in limbo or should I create a ticket?
TYOC040 cant boot debian 11 custom install installed via netboot.xyz.
boots up to grub command line.
Debian 11 template is fine.
I bench YABS 24/7/365 unless it's a leap year.
TYOC026 failure occurred: disk I/O error.
@VirMach
What is wrong with SJCZ005?
It's been a long time, what is your plan for SJCZ005?
I was actually about to post an update on it. And yes planned on moving people off a few days ago, doing that now, and we'll be crediting people for a month or extending service by a month.
Also wanted to update everyone else:
Some of these already sent out as emails, network updates, etc.
Old LAX servers that failed: All have been regenerated on Ryzen. We'll be crediting people 6 months or extending by 6 months. They still need IP addresses assigned, will most likely get that part done by Monday, couldn't get to it today.
LAXA031: Maintenance scheduled with DC. We thought it stabilized after maintenance but there was miscommunication there, it just happened to stabilize and then ran into issues again last week or so, we've scheduled they perform original intended hardware maintenance and that should hopefully stabilize it. Crediting people 1 month or extending.
SEAZ009 - Scheduled maintenance, it may fix the issue. Extending/crediting 1 month.
SEAZ010 - This one has been extended because we're having trouble getting DC hands to perform what we requested. It's "only" been about a week now but I'm completely blind on this one at the moment, we asked them to just ship it out to us so we can get the data off and people migrated sooner if they can't perform our request (I asked them to check what's being displayed last and got back "this has been completed.") No auto credit/extension if we can get this one fixed by Monday/Tuesday but you can of course request SLA credits in a ticket.
SJCZ005 - Also covered above. Trying to migrate people now. If that doesn't work I have a final idea/attempt at getting it fixed permanently as well. Migrations will be to LAX as we have limited space in SJC. Also 1 month credit/extension here.
Yeah, network's being weird but ATLZ003 was also completely going wacko in multiple ways so I had to focus most the time on that and everything else being completed today. I am seeing bursts out but I've also verified the issue you describe several hours earlier, didn't have any time to update especially since I'm trying to figure out the timeline. It does look like it was announced so it must be misconfiguration on the network equipment. I swear I cleared it though so it must have been working and they also checked on their end, we'll see.
I have VPS on SJC2005 (down) and SJC2008 (older servier, working again after being down for months, but idling and expiring soon since I don't need two VPS at that location). I'd rather not get migrated to LAX if there is some hope of SJC2008 becoming reliable.
1) Instead of migrating the SJC2005 vps to LAX could it be transferred to SJC2008, cancelling the existing SJC2008 vps, leaving just one?
2) Rather than service extensions could it be possible to have a resource increase for that remaining vps, from 768MB to 1.5GB?
If either of these vps gets migrated or transferred, it's not necessary to actually transfer the data, if it's easier at your end to just delete the old vps and create a new one that I can reinstall on. Thanks.
Ah the "remote hands with no brains attached" thing again. I fled Dallas but my Atlanta one is working well usually. And I don't need more in NY lol
This has been an interesting adventure to see just how badly some DC and remote hands setup can be. Pretty sure Virmach would describe it in slightly less interesting and more rage inducing terms though.
LAXA014 stay strong 🥲
Want free vps ? https://microlxc.net
It runs as well as my new 10gb nic does. Oh sorry, my new 10gb paperweight. sigh Here's hoping the others incoming work better lol
In positive news, I see that the new IP address for my Buffalo (NYC?) vps seems to not be blacklisted by Comcast. I hope it keeps working.
Also re:
I should add that it's ok (and expected) if the 2 year renewal cost goes to $40. I can't edit the earlier post any more to mention this.
Most remote hands techs are idiots across the board. If you need their help often or for something, most of the time it's cheaper to fly to the DC yourself or just to hire contractors. I can think of maybe 10 experiences I've had with datacenter hands that I'd consider positive, out of hundreds of interactions.
DAL008 and AZ004 network still not fixed
Followup to above: I spoke too soon, Comcast blocks the new Buffalo IP too. It's annoying. It is one of the old 384MB specials and I have a small personal website on it. I guess I can try to migrate it to another location? I'm not sure if that is supposed to work.
Does anyone know the current state of VPSShared? That's where my site was before, but it stopped working for a while due to the shared IP being null routed. I saw in the control panel that the IP changed after a while, but the new one doesn't work either.
True I guess, I haven't used them many times personally, usually just depending on either my own ipkvm or getting one connected temp.
When I think about it, I've probably been the remote hands to help out someone more times than I've used them myself
Comcrap just flat out blocks the ip entirely? Thats pretty bad even for them.
Last I remember was an update about the changeover in IP and I thought it was working again, I don't use them myself.
SJC is full due to SJCZ004 also having more serious issues, so we used all the space there to SJC. It hasn't exactly been smooth sailing for that location so there's only so much room we have left to move around within SJC.
SJCZ005 is technically more salvageable at the moment. It has to do with a loose or malfunctioning cable. It was hard to diagnose for a while because of the sporadic manner it manifested itself but I finally was able to catch it. We did have maintenance done on this node before so perhaps when they opened it up to fix the previous issues, a cable shifted and it worked fine (until it didn't.) Resource increase would be too complicated to do in bulk but I'll do it for you specifically (@willie) since you've been stuck with us in the worst string of luck ever.
Yeah I had a lot more written but realized I was going off on a weird upset tangent so I cut it out. There's a reason I'm unsurprised that such a thing would happen in Atlanta, let's just leave it there.
LAXA014 was so close to being done around Thursday/Friday. Then it crapped out mid-migration and SolusVM migration feature fully malfunctioned as a result as well, making it more difficult to do because we were doing it in batches versus a bulk script since we knew it's unstable. So the first batch was like 70% done, then it crashed, and made the system lock up. I fixed it in every way we knew possible but now we're facing another weird bug with the migration tool, so it just means we have to plan out semi-bulk manual batches instead which complicated it enough to where with everything else going on, it definitely didn't get done by Friday.
The null thing... oh god, I don't want to even get into it but I'll just say one thing: we tried. Many times.
New one shouldn't get nullrouted like that. The IP change for those also went through and there should not be any immediate problems for a long time. We just are still setting up rDNS for mailing but other than that, everything looks good and if it doesn't then it'll be an easy fix as it's not facing any weird hardware issues or anything else as of right now.
I have no idea what's going on with Comcast recently but I've had a ton of issues with them and routing from LAX office to pretty much everywhere. What's the IP start with?
Old LAX nodes that had catastrophic failure are getting their new IPs now. You can manually change the main IP to get it working sooner. I may be able to get the script working to do the main IP swaps for them but if something goes wrong that'll get done tomorrow.
(And the credits/extensions will be done tomorrow, it requires a lot of DB queries that I don't want to try to remember right now while I'm doing a lot of other things, sorry.)
Eagerly waiting for it