smallweb down and out in Los Angeles
Just a heads up for people using smallweb shared hosting in Los Angeles: a series of unfortunate events has resulted in the untimely demise of service in that location.
As per the notification email:
Dear Customer,
Earlier today the server hosting Los Angeles website data suffered a networking failure which prevented it from connecting to a network and performing standard tasks. Attempts to restore the network were made but ultimately after the server being offline for such a prolonged period of time we attempted to re-install the server and possibly migrate backups elsewhere.
However it was discovered that our system backups for this location were not suitable to be restored and thus unfortunately we are announcing the immediate closure of Shared-LAX1.
I am extremely disappointed in myself for having to deliver this news to you and this is not how I wanted 2020 to start but I would like to learn from this experience going forward to improve our other available locations.
Because we can no longer deliver your service you are entitled to a full refund. To claim a refund, please respond to this email or open a support ticket. Your refund will be processed within 7 days and your LAX service will no longer be accessible through your client area.
If you have any questions, please let me know. I am extremely sorry for this disastrous event.
Regards,
Michael
SmallWeb Ltd.
Apparently a typo in a backup script - and not enough testing to catch it.
Seems like a painful lesson learned - hopefully not too many bridges burned.
I'm personally more inclined to offer my condolences etc rather than get too bent out of shape over a glitch in a £5 per year service. And I can't help but appreciate the generally direct and transparent communication from @Mic-hael so far (compare with for example a similar situation at Gandi) ... Hopefully we'll enjoy more supportive discussion and suggestions for end-user backup best practices here. (Or whatever.)
Might also suggest service in Melbourne may be an interesting alternative for some customers, if that's a viable option. Though I'll miss the LA location - life goes on ...
HS4LIFE (+ (* 3 4) (* 5 6))
Comments
This has bitten me before. sadly, everyone learns in hard way.
https://phpbackend.com/
@PHP_Backend indeed ... I think ultimately better if these lessons get learned sooner rather than later!
Also - anyone who really needs the LA location might want to get in touch with @sanvit, who has generously offered to provide some free service (as an individual, not a provider) from their (previously idling) MyW.pt reseller account there.
HS4LIFE (+ (* 3 4) (* 5 6))
@SmallWeb 's alright- rightfully he said he will take a break from forums to focus on the business.
Sucks for those who had the LA location as their only service, hope they had backups.
OT, but man, you do have your share of hosting related stories, it seems.
blog archives
Yup, if anyone needs LA specifically, you can PM me. More details are available on this LET comment
https://www.lowendtalk.com/discussion/comment/3073629/#Comment_3073629
tl;tr
The dude put in "LUX" instead of "LAX" in setting.
He never checked / verified that backup was there.
But then, considering its price, I wouldn't feel entitled if I were a customer.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
I heard he started a land war in asia - and went against a Sicilian when derp was on the line.
HS4LIFE (+ (* 3 4) (* 5 6))
I would. The price providers ask me to pay is to provide the service they offer. This whole, because it's such a small amount being justification is bollocks quite frankly. I mean not even checking the backups are functioning..
That said, your data is only as safe as the backups you take and check.
At least a full refund is being offered.
A buck or 2 a year service is not something I'd bother ranting about.
At the same time, a host being quite professional raises expectations, so basically he dug his own grave. In that sense, I think @cociu_with_sisters is doing a good job keeping expectations in line.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
It's not, but then that is not the point. It is the justifying of an avoidable issue like this with "it only cost.." is not acceptable in my view.
Backups at a provider should never be seen as your only route to recovery but;
Provider backups mean restoring all client accounts after a failure saves them time even if they do have their own.
It could also be that the provider has a newer backup point which saves the client re-work.
The most important take away here is that if the provider didn't even notice the backups were not happening in one location then they are not even being tested in any other location so assume they don't exist at all.
Promises are easy to make with words. Delivering them is different no matter what the price point a product has.
That's probably why many (don't know exactly how many) are pissed at politicians because barely anyone keeps their words once elected.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
"entitled" may be a somewhat loaded term (lol) ...
as far as reasonable expectations go well I dunno ... hope for the breast, plan for the wurst?
and if your data matters to you ... then you're already doing your own backups offsite in a serious way, ammirite?
I am a bit surprised to see service in a given location discontinued altogether - though this does seem to happen more often with new-ish providers offering low-priced services based on other low-priced providers, so not exactly shocked to see happening in this case. But I am disappoint. Maybe some room for improvememt in managing this sort of worst-case scenario going forward. Maybe not.
I'm guessing that once the dust settles we might get an incident report with a bit more detail and maybe even some (hopefully diplomatic and professional "with all due respect") backstory about the network issue or other motivations to abandon ship in LA. Or not ... I'm certainly not paying enough to expect more than "best efforts" all around, but hope springs eternal so ¯\_(ツ)_/¯
HS4LIFE (+ (* 3 4) (* 5 6))
The thing with shared hosting is that usually, it is a pita to keep constant automated backups and/or incremental ones. Backup and transfer whole cpanel accounts in another location cannot be done easily in most of shared providers, especially if there are not installed external plugins and/or there is not any jailed ssh access to make life easier...
For a small web site with minimum to none updates, though, it's easy to back it up and download it locally just in case. The worst scenario is with email accounts, as the only viable option for constant backups without ssh access, is imapsync - and for that, you have to have access to a server with root privileges to install it...
Of course, if you have valuable data, then, you will not host them in a 10$/y shared cpanel hosting without more than one backup scenarios
If backups are essential for a customer, it is 100 x essential for a provider... I do not have services with @SmallWeb , and Mic-hael seem a really nice guy, but the structure of his shared environment as exposed due to the failure, is extremely poor. A single backup with a custom script that had never being verified if worked? Not even a simple check that the size of the tarball archive was the same in destination as in the original server? It's not about the money of the affected services, it's just ringing bells on the whole structure of his servers.
Also, he claims that there was a network failure that was difficult to be restored in a proper time window and he tried to restore services to another location due to that, when he discovered that the backup was not valid. There is a plot hole here: if there is an issue with networking, he could try to resolve it much more after he discovered that the backups were none. Why did he decide to abandon a server (or, better, a whole location!) that -as he claimed- the issue with it is networking and not corrupted disks / raid etc. Smells like he has a fight with the server provider and he is not able to get access to the node...
Anyway, the statement that it is a big failure from his side, the full refunds and the apologies to his customers can reduce the bad reputation that such an incident creates. And his claim that he will learn from his mistakes to make a better infrastructure and backups, is totally valid - only such a failure can really make someone to triple-check the next time everything.
• If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
• If such a program has not crashed yet, it is waiting for a critical moment before it crashes.
I still go with SmallWeb when I need shared hosting. The best support ever!
Lesson learned, SmallWeb will be stronger!
@jvnadr - that is some insightful analysis. Thanks for sharing it here.
I hesitate to speculate too much about exact motivation for "abandoning ship" in Los Angeles but at this point I would just say it is a big question mark for me too. Hopefully we'll get some clarification on this after some time has passed - I think it's wise to recognize how much stress can distort perspective when the "downtime" clock is still ticking.
As for expectations about learning ... there's aptitude, and there's attitude. The attitude often comes from experience. And experience sometimes comes from making mistakes. So this may be the proverbial cheap tuition in the school of hard knocks.
HS4LIFE (+ (* 3 4) (* 5 6))
It all just means the end is always right around a corner and you guys should join Nigh sect.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
Definitely.
Anyone who needs help please let me know.
@SmallWeb this too applies for you mate. Feel free to PM me if you need help or if you think there still is a chance to recover the data.
Such events happen in our industry. We always advise our clients to backup their data.
I have upgraded both of your lifetime reseller accounts to the double of diskspace free of charge so you can help our forum members.
Thank you for the messages.
As Lee says, the price point can not be an excuse. I never offered services with the upfront statement that "it's cheap so don't have any hopes". The results of my actions led to the integrity of the service being damaged.
For @jvnadr
The testing of the backup script was abysmal and did not cover all locations. Since this, it has been validated for all remaining locations.
Regarding the server, there's no correlation with the provider in question. No upstream support was offered and agreed to by me when I launched the location. They have a very respectable service that I aspire to be even half as good as.
The closure was extremely abrupt. I had not thought about starting from fresh, this is my first experience of such a catastrophic failure and my first instinct was to inform customers their data was gone. I am overlooking potential options regarding this.
Michael from DragonWebHost & OnePoundEmail
@SmallWeb
Sometimes you're the guy stuck in the barrel. Other than offering a full refund, there's virtually little else you should be required to do. Yes, it would be nice if there were other options, but sometimes shit happens and it all goes wrong.
You seem to have accepted it with grace, humility, and hopefully don't get too many chargebacks-out-of-spite from this.
"It's a hard life- to be a stick insect." - Karl Pilkington
Of course, you can either blame Trump or Brexit. Or even both.
Some seem to blame those two for everything that is apparently wrong in their worlds.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
Well that's a good reminder for me to verify some backups
🦍🍌
Dicks out for HaramLA.
"It's a hard life- to be a stick insect." - Karl Pilkington
As a continuation of my last point. The server has now been rebuilt and configured. I have reprovisioned 99% of LAX services bar those who have been refunded or transferred already. Those individuals will be contacted seperately.
Please expect official communication about this.
Michael from DragonWebHost & OnePoundEmail
Nexus Bytes Ryzen Powered NVMe VPS | NYC|Miami|LA|London|Netherlands| Singapore|Tokyo
Storage VPS | LiteSpeed Powered Web Hosting + SSH access | Switcher Special |
LET, LES, HB or other, it is truly the community that keeps this industry alive and motivated. As vyas mentioned, I will be taking a leave from the forum end of things for a while to focus on correctly building my business. This was not something to be taken lightly, so of course I will be available to answer questions regarding this.
Michael from DragonWebHost & OnePoundEmail
Ouch. Happens I guess
I'd sorta expect the backup software to have verification included? e.g. I know one of the drives in my NAS is bad because duplicati has been complaining about hashsums failing (It randomly picks one archive to verify on each run)
Mistakes do happen, sorry if I was too harsh earlier. After all, this was a good lesson to avoid such mistakes in the future, as I said before. And, many of LE providers would not approach the issue like you but they would just say "hey, there was a catastrophic issue in a server, your data was lost, sorry but after all, it's clients responsibility to take backups, so, f** off"! Your attitude on this makes me think that this first big failure will turn you into a better provider.
If you can share it atm, understanding that there are bigger priorities now: Apart from the backup failure, what was the catastrophic issue on the original server that caused the data loss and made the node impossible to be restored ?
• If a program actually fits in memory and has enough disk space, it is guaranteed to crash.
• If such a program has not crashed yet, it is waiting for a critical moment before it crashes.
If I were to go through the process again, I would have never overwrote the data (regardless of backups) until a possible solution was found to preserve all original data. My original intention was to set up a second server, in which I would have discovered the backup issue however the timing required for relicensing and reconfiguring IPs (assuming backups were available) were my main concern in my wrong split decision to overwrite.
During the time spent trying to solve the network issue, I did multiple tests and reconfigurations of the network interfaces and discovered missing commands and thus packages which make the network function correctly. The downtime at this point and after further attempts was already disappointing, leading to that wrong split decision.
I don't take your comments to be "too harsh" at all. I think all of the opinions on this matter are vital. Thank you.
Michael from DragonWebHost & OnePoundEmail
Smallweb stays on The Whitelist. That's my endorsement based on a reasonable assessment of the issue and its handling.
However, it is a good time to take stock of everything, (not just technical) because the next issue may not be met with such warm reception, depending on the nature of the issue (fried switches are understandable).
Deals and Reviews: LowEndBoxes Review | Avoid dodgy providers with The LEBRE Whitelist | Free hosting (with conditions): Evolution-Host, NanoKVM, FreeMach, ServedEZ | Get expert copyediting and copywriting help at The Write Flow
When you have only serial console access and no network, but you need some packages, it's possible to "upload" files via console.
catto paste the Base64 into a text file on the server.base64command on the server.If you have lost
catorbase64on the server, you can useecho -nebut it's more difficult.We accept Karma donations for the last flan. 🍮 affbrr
My personal point of view (not sure if it's shared by most customers, or not):
1)
Only those who never do anything make no mistakes. The difference between the good and the bad providers is whether mistakes are something that happens regularly, or a rare exception. Also, it matters if measures are taken to prevent the same mistakes from re-occuring.
For some reason, many people / companies fail to test restoring from their backups. Similar goes for other safety measures. To me it's like cancelling an emergency evacuation exercise because of rain.
It's better learning from other people's mistakes, than from one's own, but only the really stubborn people don't learn from their own mistakes at least.
2)
Again, in my point of view: longer downtime is best announced, though explaining the reasons once it's already started is also fair.
Either way: it's better to have a longer downtime on occasions, knowing that things are fixed "systematically", for better functioning in the long run - than it is to have downtime (and/or data-loss) every now and then.
3)
Emergencies are wars, disasters, violent crimes - in IT, there's only poor planning - generally speaking.
Customers who need (near) 100% uptime should have failover website deployments in more than one geo-location. And backups - that's always necessary.
Still, many (most?) people are not reasonable. They don't want to spend the money it takes for the above mentioned, but they want to be convinced that a provider (whichever they choose) will provide 99.999% uptime.
One thing is having a TOS that says that backups are a convenience but no guarantees/responsibility are taken. With explanation that going below 99.99% gives a few dollars compensation (depending on the downtime duration), up to the price of monthly subscription.
Another thing is stating that clearly, on the front / sign up page. Don't think one would sell a lot of accounts with such policy.
Which all results in expectations too high.
4)
Amount of money spent on hosting makes no difference and is no guarantee. Not really.
Educational situation with highly reputable and rather expensive MDDhosting (TLDR: all servers down and all backups messed up, week of downtime):
https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/
Gitlabs experience with not having tested restoration from backups prior to trying it in production, when it was needed:
https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/
As a customer - I'm aware that I shouldn't expect really good service for dirt cheap. Aware that many people promise many things, not taking everyone's word for granted.
As a service provider - regardless of the price, I would require myself to offer what's promised. Think in the long run it's all about reputation and trust. Would probably never get rich, but if I were to sell crap (not claiming, nor implying that Smallweb is doing that), I'd state it up front. MXroute email provider is a great example of that. Their website clearly states, up front, something along the lines (paraphrasing from memory): "we offer cheap service that works well, BUT we cut the costs on support - you won't get much help with the basic stuff and for most questions will be lead to a community support forum, if you want more hand-holding, pay a lot more with other providers". That's fair and straight forward from the start. Though I'm sure it turns many customers away initially, the customers that do take the service don't get disappointed, quite the contrary. They could have promised all bells and whistles on the front page, but it would have backfired (one disappointed customer review can hardly make up for 1000 good reviews).
Whoever has read this till the end - come to Serbia, you've earned a bottle of home-made rakija to re-set your brains...
🔧 BikeGremlin guides & resources
Lax Smallweb is up. Reborn though not reincarnated.
From mail received:
Login details yada yadaa
blog archives
@yoursunny and @bikegremlin thanks for adding those thoughtful comments!
And
for everyone who thought twice and decided to save the potential drive-by shitposts for some other thread. I think the real lowend spirit may shine a bit brighter here for your forbearance. 
@SmallWeb I hope this serves to confirm a solid level of support from your customers and colleagues alike. Re5pekt! Maybe, just maybe you have earned enough of it already by "paying it forward" - I'm not sure exactly how, but that does seem to be what we are seeing here.
As a startup, and as a solo practitioner, you may often feel like you're way out on a limb over thin ice without a net. So when something goes wrong, a sense of urgency verging on panic can easily lead down a slippery slope to an avalanche of errors. If you know in those situations that you can reach out for advice - technical and otherwise - then that's a much better place to be in.
(I'm about to murder more metaphors but will stop short of suggesting to take a deep breath when the shit hits the fan.)
Really glad to see the Los Angeles location is now rising from the ashes. Will happily leave you to it now. Cheers!
HS4LIFE (+ (* 3 4) (* 5 6))
Suggest to make backups a beta feature , free out of goodwill, instead of part of advertised service. It will save Ur own ass and manage £1 expectations as well .
I bench YABS 24/7/365 unless it's a leap year.
Hosting Package: ½GB
Domain: *****
Package Price: £3.00 / Year
from £11.00 / Year to £3.00 / Year
@MikePT this correct ?
Thank you
@SmallWeb :-)
blog archives
typo
on single day 11.11 does anyone remember the promo ?
is this for 1 year? how many years for package ½GB, sorry I forgot
https://www.lowendtalk.com/discussion/comment/3040912/#Comment_3040912
https://www.lowendtalk.com/discussion/comment/3040916/#Comment_3040916
https://www.lowendtalk.com/discussion/comment/3041003/#Comment_3041003
I think these were the two three offers.
p.s: I read the entire thread and wow, some good offers in that.
blog archives
Wow amazing, for package ½GB (£1.11/Year ) = 10 year (£11.11),
Thank you
Some great advice and lessons to be learned. Thank you all. See you in Serbia @bikegremlin
@isunbejo By the looks of it that was the £11.11 for 10 years promotion. All plans have been converted to their yearly equivalent.
I just wanted to post the aftermath email that should currently be on the way to affected customers.
Thank you.
Michael from DragonWebHost & OnePoundEmail
No problem, my data backup still exists in a safe place.
production -> ftp / rsync <-storage box -> gdrive -> myhome storage
now it has been restored and running well
Thanks.
I just want to thank all of the members of the forum who reached out to help Smallweb. That's what makes a community work and LES a nice place to be.
.> @vyas said:
This is good for anyone that uses the WordPress UpdraftPlus plugin for backups. I reckon that this is the most popular (perhaps not the "best') backup plugin, but it only restores to the original host. If a host deadpools, the UpdraftPlus backup, stored at Dropbox, Google Drive, etc is useless. Only the paid version supports migration between hosts.
So good that LA is back up, and I hope that UpdraftPlus restores work for those impacted. I use the NYC datacenter and working great.
What is your best WP backup scenario
@SmallWeb proud to see you back mate, I knew you'd do it.
I consider using a plugin for backup a poor solution. Not expecting an application to back itself up.
Prefer to set backups with the hosting server - to a remote storage of choice.
DirectAdmin backup.
cPanel backup with JetBackup (to supercheap Backblaze B2 storage).
🔧 BikeGremlin guides & resources
Don't use WordPress.
Static site generators are better because all your contents are safely on GitHub before it even reaches your hosting server. Your local git clone and the GitHub repository are backups of each other.
Don't forget to commit NGINX configure file into git as well.
If your host deadpools, rebuild the site from git. It only takes 30 minutes.
We accept Karma donations for the last flan. 🍮 affbrr
It depends on the use case - type of site, user's knowledge, experience...
🔧 BikeGremlin guides & resources
Yeah, I can easily imagine a newbie figuring out how to commit and clone for more than 30 minutes ;-)
Clouvider Limited - VPS in 11 datacenters - Intel Xeon/AMD Epyc with NVMe and 10G uplink! | Dedicated Servers
When I build a site for a client, I let them email me every article they want to post, and then charge them per thousand characters and per picture upload. If they want pictures inserted in the middle of article instead of at the end, it's a surcharge.
This was 2006. There was no git. I login to my CMS and paste the Word file, then insert each picture. The CMS generates static HTML files, and uploads them to another FTP server (the web server of another client) for backup.
The client could use this CMS themselves, but they'd rather pay the per article charge.
I'm sure they aren't using this CMS anymore. I wrote the CMS and sold them a 10-year license that expired in 2018.
We accept Karma donations for the last flan. 🍮 affbrr