@corbpie said:
Damn that’s probably my longest serving service gone, had it since 2016
Bought my dedi on 08/15/2016, so same. I still have a few VPS with other providers that are older but definitely the longest dedicated server of mine by far.
@skorous said:
Question: Are the emergency dedi's already racked and if so are they at multiple locations or all in LA for example? Tangential to that, how will location of replacements be handled?
First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately available, just waiting on us to sort it out and send details. Around 15% are being worked on immediately and most likely delivered to us today. Around 30% more have been prioritized to begin delivering tomorrow. These are all figures I'm vaguely quoting from our new provider.
The specifications will vary. IPv4 will be less for some people. Some will be 16GB, others 32GB (similar to what we provide.) They'll also be HDD and SSD setups. We were supposed to send in SSDs, this got delayed on my end (to try to make more of them SSD.)
@VirMach said:
We do have some tickets from before we sent out the emergency notice
Where/how are these being sent? I haven't received anything via email and I see nothing in the billing panel related to this.
Had I not checked the forums, I would have no idea of the severity of the outage.
I had to send these via WHMCS, which we usually don't do. It's listed as 100% sent on one screen, but I realized on another screen it's listed as "remaining" with "no failed emails." None of it makes sense, it's possible these actually didn't send which would be horrible as the emails were very important and I did my best to send them quickly.
Checking further to see what happened in the logs.
[Emergency] Potential Dedicated Server Service Disruption
Dear VirMach Customers,
We are sending this emergency notification to let you know of a potential service disruption in regards to your dedicated serve0(s.) We will discuss additional details when possible, but at this time, it is important to ensure your backups are up to date.
Please ensure you maintain a copy of all important files in a secondary location, outside of your dedicated server. This is always best practice, but there is currently an elevated risk of service disruption.
If there is an unexpected interruption, please contact us in the "priority" department and mention in the subject that the matter is regarding your dedicated server with us. We will continue to provide SLA credit per our terms of service. We have also prepared replacement servers which we will be distributing on an emergency basis. If you would like to more immediately switch your dedicated server to a server located with our new datacenter partners, please create a ticket in the "priority" department called "Dedicated Server Migration" and provide [1] your existing dedicated server credentials, [2] your minimum requirements for your new dedicated server, [3] your desired location(s) for your new dedicated server, if available.
Thank you.
Oh man if this actually didn't send after listing as sent ... that really sucks. Because it would have been helpful.
Shows up as such on WHMCS, but checking email logs definitely doesn't show anything sent around that time. This is quite upsetting.
Sending report shows as:
Client Failure Reason
There are no failed emails
But also shows as:
184 Remaining
Looks like we're going to have to just create individual tickets at this point, I don't know if wee can trust the email tool to re-try at this point. I know all other emails are sending though so this definitely has to do with the specific bulk email tool.
Forced the cron through again, same status. Forcing other crons, nothing. I had to specifically force the --ProcessEmailQueue cron after forcing the --EmailCampaigns cron. They have a cron that updates the status without actually doing it, and then another cron that processes them.
It might still not send but now it's listed as 25 Sent / 159 Remaining (and on other screen still says 100% done.)
It's actually just stuck at 25 now. I might have to force it 6 more times?
Edit -- 30 sent, 34 sent... yeah I actually have to force it once per 25 emails because it apparently does max batches of 25.
Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
Edit -- Fourth batch being forced.
Edit -- Fifth batch being forced.
Edit -- Sixth batch being forced.
Edit - Seventh batch being forced.
Edit - Eight batch being forced.
Edit -- All showing as sent on WHMCS but it's basically showing the same thing 184/184 [100%] so I'm going through mail logs and recounting them.
Man, it never seems to stop for you! Good luck with it all, and remember to rest and sleep in the midst of this. At the end of the day, they're only servers, your health is more important.
@VirMach said: We're not going to use this situation to get out of any previous obligations, and that includes "FreeMach." You'll have a replacement. It'll be the same price, so free if it's free.
@VirMach said: Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
That's actually so crushing knowing those didn't go out 5-6 hours ago.
Ouch, failed by yet another thing not really in your control. Did you like smash some mirrors or walk under a truck full of ladders or something lately?
I can only hope that karma pays you back for all of this hell you've been goign through with many years of things working well. And maybe like a bathtub full of whiskey or something after this..
@Jab said:
Jesus this WHMCS, I am shocked that they don't have a competition...
I wrote out this insane rant about them the other day, and then I realized how insane it sounded and deleted it (I think.) It was essentially a 10 paragraph huge clump of text. This was after a period of not sleeping for several days. But the shortest way I can re-iterate the rant is that it is the single piece of software I despise the most, not because of the DRM or bad coding, but simply based on how ridiculous the logic is behind each action.
@VirMach said:
We will of course continue to honor everything that's within our control. We are still planning on continuing as expected with the migration specials. If your friend would like to take that route, then just ensure they request a replacement server. If the old machine is cancelled, as a direct result of today's events, I will consider allowing them to continue the migration end of the special, although it's not currently set up to be processed in that manner.
The only changes we're offering are meant to be to the customer's benefit. We're not for example, going to cancel a special offer for a dedicated server at a good price if the customer wants to continue it on a new server with our new datacenter partner.
First, wanted to echo the sentiment of understanding, I know you've got to be WAY more stressed about the situation than I am. So, I'm happy to wait for a replacement server in another datacenter, and don't need access to the old data, I've got recent enough backups. I'd rather not end up with a refund / canceled server from all of this, should I just create a priority ticket and explain the same there or how are you wanting people to handle this? I'd like to make it the least work on you / your team, since I know you've got plenty of it already.
all the best Virmach. personally started off with very low expectations on the quality of your VPSes especially with heavy MJJ userbase and previous association with cuntcroissant.
seeing your struggle over a year (or more) and persistence on fixing all these endless problems at such a huge scale with all these naysayers and backstabbers, i am now rooting for you to win.
not exactly a happy customer right now, but i want you to win and win big so i can be a happy customer on dirt cheap ryzen VPSes.
P.S. do plan on longer term support levels when this over but who knows when it will be
@VirMach said:
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end we can control. Then, first come first serve basis on emergency tickets created requesting a replacement server, we'll begin provisioning those today, as well as backup servers for anyone who requires it. Both of these would temporarily have no charge (as in, free for the time being.)
This should give everyone a few different choices given the scenario, so can they proceed with a purchase as necessary elsewhere, or to at the very least have some VPS or server to hold any backup or temporary home.
Let me know if you guys have any questions or suggestions but please keep in mind they'll have to be specific and direct.
is there any chance i can get my data back?
Are you on a dedicated server that's currently offline?
Successes usually come to those who are too busy to be looking for it. Ordinary people are satisfied with their short term achievements, but an extraordinary person continues working hard and accomplishes goals one after another. You are one of the extraordinary people who is worthy of their success. Best wishes for VirMach's continued success in the future.
some may claim milllions lost, others will say billions gained thanks to patience.
on a lighter note, we will celebrate with a thread full of bus images once this is over. If @FrankZ consents and participates.
@VirMach my buffalo server since 2018 went down today had been very satisfied since so I certainly hope to get a replacement.
Its a bummer but oh well nothing mission critical since I have my backups.
@VirMach said:
Hivelocity's the one that acquired Incero right? I don't know why they didn't come to mind (HV) when I was thinking about Dallas. We can't do it right now but there's absolutely no way we can stay with Flexential at this state, WTF.
You've got dozens of options in Dallas. The "go-to" is Carrier-1 but there are many others, DallasColo is another example, or Stack.
Where you are now, the network reliability is crap. The whole connection goes down every week or two and takes out VirMach, DediPath, Nexril, etc. I literally do have better network reliability at my house.
@Daevien said:
Yeah, know anyone in Dallas with a rack in their house? They'd be more useful & have better uptime probably.
This is actually true.
$ uptime
01:15:06 up 229 days, 13:53, 1 user, load average: 0.00, 0.00, 0.00
$ ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=4.87 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=2.95 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=4.92 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=3.05 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=5.84 ms
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4012ms
rtt min/avg/max/mdev = 2.957/4.330/5.848/1.139 ms
Comments
Bloody hell, I called it right.
Best of luck sorting this mess out and somehow try to get some rest.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Damn that’s probably my longest serving service gone, had it since 2016
Where/how are these being sent? I haven't received anything via email and I see nothing in the billing panel related to this.
Had I not checked the forums, I would have no idea of the severity of the outage.
Bought my dedi on 08/15/2016, so same. I still have a few VPS with other providers that are older but definitely the longest dedicated server of mine by far.
First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately available, just waiting on us to sort it out and send details. Around 15% are being worked on immediately and most likely delivered to us today. Around 30% more have been prioritized to begin delivering tomorrow. These are all figures I'm vaguely quoting from our new provider.
The specifications will vary. IPv4 will be less for some people. Some will be 16GB, others 32GB (similar to what we provide.) They'll also be HDD and SSD setups. We were supposed to send in SSDs, this got delayed on my end (to try to make more of them SSD.)
I had to send these via WHMCS, which we usually don't do. It's listed as 100% sent on one screen, but I realized on another screen it's listed as "remaining" with "no failed emails." None of it makes sense, it's possible these actually didn't send which would be horrible as the emails were very important and I did my best to send them quickly.
Checking further to see what happened in the logs.
I'll provide a copy here, typo included.
Oh man if this actually didn't send after listing as sent ... that really sucks. Because it would have been helpful.
Shows up as such on WHMCS, but checking email logs definitely doesn't show anything sent around that time. This is quite upsetting.
Sending report shows as:
But also shows as:
Looks like we're going to have to just create individual tickets at this point, I don't know if wee can trust the email tool to re-try at this point. I know all other emails are sending though so this definitely has to do with the specific bulk email tool.
I never received that notification
Forced the cron through again, same status. Forcing other crons, nothing. I had to specifically force the --ProcessEmailQueue cron after forcing the --EmailCampaigns cron. They have a cron that updates the status without actually doing it, and then another cron that processes them.
It might still not send but now it's listed as 25 Sent / 159 Remaining (and on other screen still says 100% done.)
It's actually just stuck at 25 now. I might have to force it 6 more times?
Edit -- 30 sent, 34 sent... yeah I actually have to force it once per 25 emails because it apparently does max batches of 25.
Edit -- Third batch being forced now. At this point it's completely my fault for not expecting this.
Edit -- Fourth batch being forced.
Edit -- Fifth batch being forced.
Edit -- Sixth batch being forced.
Edit - Seventh batch being forced.
Edit - Eight batch being forced.
Edit -- All showing as sent on WHMCS but it's basically showing the same thing 184/184 [100%] so I'm going through mail logs and recounting them.
Edit -- Logs for the most part look correct.
Man, it never seems to stop for you! Good luck with it all, and remember to rest and sleep in the midst of this. At the end of the day, they're only servers, your health is more important.
Just got it, boss
Head Janitor @ LES • About • Rules • Support
That's actually so crushing knowing those didn't go out 5-6 hours ago.
I just got notification also .. thank you
Priority Ticket Created #998050
Jesus this WHMCS, I am shocked that they don't have a competition...
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Ouch, failed by yet another thing not really in your control. Did you like smash some mirrors or walk under a truck full of ladders or something lately?
I can only hope that karma pays you back for all of this hell you've been goign through with many years of things working well. And maybe like a bathtub full of whiskey or something after this..
CPANEL and WHMCS, two things I definitely don't miss about the hosting world lol
I wrote out this insane rant about them the other day, and then I realized how insane it sounded and deleted it (I think.) It was essentially a 10 paragraph huge clump of text. This was after a period of not sleeping for several days. But the shortest way I can re-iterate the rant is that it is the single piece of software I despise the most, not because of the DRM or bad coding, but simply based on how ridiculous the logic is behind each action.
When did WHMCS get logic? It was just random shit stacked higher and higher from what I remember lol
@Virmach GET SLEEP!
(Silly Sod)
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Aha, email came through.
First, wanted to echo the sentiment of understanding, I know you've got to be WAY more stressed about the situation than I am. So, I'm happy to wait for a replacement server in another datacenter, and don't need access to the old data, I've got recent enough backups. I'd rather not end up with a refund / canceled server from all of this, should I just create a priority ticket and explain the same there or how are you wanting people to handle this? I'd like to make it the least work on you / your team, since I know you've got plenty of it already.
all the best Virmach. personally started off with very low expectations on the quality of your VPSes especially with heavy MJJ userbase and previous association with cuntcroissant.
seeing your struggle over a year (or more) and persistence on fixing all these endless problems at such a huge scale with all these naysayers and backstabbers, i am now rooting for you to win.
not exactly a happy customer right now, but i want you to win and win big so i can be a happy customer on dirt cheap ryzen VPSes.
P.S. do plan on longer term support levels when this over but who knows when it will be
I bench YABS 24/7/365 unless it's a leap year.
yes,in DAL and LA
Successes usually come to those who are too busy to be looking for it. Ordinary people are satisfied with their short term achievements, but an extraordinary person continues working hard and accomplishes goals one after another. You are one of the extraordinary people who is worthy of their success. Best wishes for VirMach's continued success in the future.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
@VirMach,
Keep chugging-
some may claim milllions lost, others will say billions gained thanks to patience.
on a lighter note, we will celebrate with a thread full of bus images once this is over. If @FrankZ consents and participates.
blog | exploring visually |
@vyas - I'm in and I am sure many others will join in the fun.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
@VirMach my buffalo server since 2018 went down today had been very satisfied since so I certainly hope to get a replacement.
Its a bummer but oh well nothing mission critical since I have my backups.
All the best @VirMach
Was wondering if we should have a front page post separated from this thread so no "DEDPOOL" threads spring up?
Cloudron.io - Lovely self hosted management suite [aff]
HetrixTools - Highly recommended uptime monitor [aff]
You've got dozens of options in Dallas. The "go-to" is Carrier-1 but there are many others, DallasColo is another example, or Stack.
Where you are now, the network reliability is crap. The whole connection goes down every week or two and takes out VirMach, DediPath, Nexril, etc. I literally do have better network reliability at my house.
This is actually true.
I have a question, @VirMach, do you even sleep? You work damn hard from what I've gathered so far in this thread, jesus.
I can't even begin to fathom the volume of tickets that you've probably gotten so far.
I am a representative of Advin Servers