Hosthatch - partial Chicago downtime

root · March 2022

Anyone else experiencing the downtime in Chicago?

@hosthatch - it seems there is some weird error making the storage VPS not boot.

Error displayed at boot: "XZ-compressed data is corrupt. System halted."

kheng86 · March 2022

I have been having short period of downtime every now and then (sometimes, everyday) for the past 6 months with my Chicago VPS. All other VPS from HH were fine. I have asked for a migration to other location, but was refused because it's part of a "bundle".

tetech · March 2022

Had a similar experience with LAX storage a while back, it was a bad RAID card. The end solution was that I got set up on another node and rsync'ed the data across. I ended up losing some % (maybe 5%) of the data due to corruption, which I had to restore from another location.

The point of writing this message is to say, even if your VPS comes back up normally, do a check of the filesystem for corruption.

FrankZ · March 2022

From OGF

Mike W — HostHatch Representative
Hello,

Apologies for the delayed response.

We're still investigating the ongoing issue affecting the node STOR4.CHI.

The node had disk errors and was hard rebooted, causing most VMs to have errors when booting up.

We are still doing our best to restore all VMs back to their working state.

If it is possible for you, we would recommend that we spin up a new VM for you with the same specifications and boot up RescueCD on the current/affected VM. You can then move over the data to the new VM. We can also move over your old IP address to the new VM if needed. Please let us know.

root · March 2022

Out of nowhere, without any ticket or message, I get an invoice for a new service in the new cloud portal.

@hosthatch - care to explain?

FrankZ · March 2022

@root said: Out of nowhere, without any ticket or message, I get an invoice for a new service in the new cloud portal.

I got a bill as well. Does the new service they spun up for you have different specs?
Before I had 3950 MB RAM, 2147.5 GB HDD, 6GB transfer.
The new one is 1837 MB RAM, 1999.5 GB HDD, 10TB transfer.

root · March 2022

@FrankZ said:

@root said: Out of nowhere, without any ticket or message, I get an invoice for a new service in the new cloud portal.

I got a bill as well. Does the new service they spun up for you have different specs?
Before I had 3950 MB RAM, 2147.5 GB HDD, 6TB transfer.
The new one is 1837 MB RAM, 1999.5 GB HDD, 10TB transfer.

What is even more interesting, is that there is no clear information in the new cloud panel about server specs, when the new service expires, when it will renew, or what will be it's renewal price. The new cloud panel is not useful because important information is not displayed so customers can be clearly informed about what they bought.

This, combined with loss of data, poor support, and forcing people on a new panel (because everybody will be migrated in the following weeks as it is announced) makes me think twice from now on when it comes to HostHatch. It was good though, while it lasted.

@hosthatch - If it works, don't fix it !!!

FrankZ · March 2022

Ok, so it is not just me that can't find any billing information in the new HH panel.

cablepick · March 2022

@root said: because everybody will be migrated in the following weeks as it is announced

That message has been there since November when they opened the new panel for Black Friday. I asked last week about a migration path for existing services and they responded saying no method or timeline right now.

@root said: no clear information in the new cloud panel about server specs

They recycle plans on the backend so even in the old panel the information about server specs changes as they update the plans for new sales. At least you can see how much it cost and when it's next due. My crashed storage server from this episode is still pending in the new panel so I've haven't explored it too much and I didn't buy any new services this past black friday.

root · March 2022

@cablepick said:
My crashed storage server from this episode is still pending in the new panel so I've haven't explored it too much and I didn't buy any new services this past black friday.

I did not buy on BF2021 either, and don't plan on buying more. If they have a beta cloud, why force customers into it, especially after bad moments like these when data gets lost? I want transparency, and I want to be clearly informed about what I buy or what service I have. I don't like how they treat their customers in support with long waiting times, or the way things are going.

root · March 2022

It's funny. I don't know how am I supposed to transfer an encrypted partition of 2,047,999 megabytes onto a new server with maximum 1,906,911 megabytes.

Even though this should be a migration, in my case it is actually a downgrade with a difference of 141,088 megabytes (more than 137 gigabytes). @HostHatch forgot how in low-end market, every gigabyte counts, especially when one needs to migrate their data as an emergency.

This is why I ask for transparency and clear information inside the VPS panel.

cablepick · March 2022

@root said: I don't know how am I supposed to transfer an encrypted partition of 2,047,999 megabytes onto a new server with maximum 1,906,911 megabytes.

I believe the old panel used GiB (1 GiB = 1024 MiB) and the new one uses GB (1GB = 1000MB). Once my pending Chicago server on the new panel gets activated I can compare it to my LAX one on the old panel and report back.

root · March 2022

@cablepick said:

@root said: I don't know how am I supposed to transfer an encrypted partition of 2,047,999 megabytes onto a new server with maximum 1,906,911 megabytes.

I believe the old panel used GiB (1 GiB = 1024 MiB) and the new one uses GB (1GB = 1000MB). Once my pending Chicago server on the new panel gets activated I can compare it to my LAX one on the old panel and report back.

My "new" Chicago server has 1,999,541,108,736 bytes. So actually @hostatch believes that 2,000 million bytes is the same thing as 2 Terabytes (which was in fact advertised). What I got now is not what I bought, and the specifications are not displayed at all in the new panel.

skorous · March 2022

I believe Seagate redefined what a megabyte back 20-30 years ago and the rest of the industry followed suit shortly after.

FrankZ · March 2022

I'll keep my comments short and calm here. I received a new email this morning that HH unsuccessfully tried to charge my c/c for the now overdue invoice of the new storage server. Still no response to the ticket.

skorous · March 2022

@FrankZ said:
I'll keep my comments short and calm here. I received a new email this morning that HH unsuccessfully tried to charge my c/c for the now overdue invoice of the new storage server. Still no response to the ticket.

Man, this is messy.

ehab · March 2022

@root nice to see you

root · March 2022

@ehab said:
@root nice to see you

Nice to see you too.

FrankZ · March 2022

From the OGF

@hosthatch said:
Sorry for the silence on this. We're doing our best to get everyone back up and running after a storage node had a RAID failure in Chicago.

If you got an invoice that you shouldn't have gotten, please ignore it, it will be canceled automatically. It won't result in a service suspension. We're setting up new VMs for the affected customers and allowing everyone to boot up rescue/live OS on the old VMs to move the data to the new VMs. Some of the new services had invoices generated by mistake, and those are (or will be) canceled.

life · March 2022

I am done with hosthatch. I have paid around $1100 in last year for multiple servers with them from 1 to 3 year engagement. They are not getting a single penny more from me.

beagle · March 2022

Maybe I've been lucky. I have storage servers running in LA, UK and NL for the last 3 years and they have been solid so far. All BF offers.

The few times I needed support for upgrades due to BF offers they indeed were slower than expected. But overall still value for money for backups.

They said they were investing in support to improve service levels but it seems users are not seeing it yet.

deank · March 2022

I dare say downtime is a good thing.

Without downtime, you will never appreciate uptime.

skorous · March 2022

Hosthatch does three year deals?

nfn · March 2022

I have 4 NVMe servers with them without any problems.
The only downside is response time that eventually should be solved in a long run.

Not related to HH, but we must always have backup plans and redundancy.

Saying that, I'm ready to get some more @hosthatch servers

root · March 2022

@nfn said:
Not related to HH, but we must always have backup plans and redundancy.

Saying that, I'm ready to get some more @hosthatch servers

So it is not related to HostHatch, and yet it is related to HostHatch?! Maybe I should wish you good luck in buying more (I guess), since you're already aware you'll need luck when you rely on your own backups for production.

nfn · March 2022

@root said:

@nfn said:
Not related to HH, but we must always have backup plans and redundancy.

Saying that, I'm ready to get some more @hosthatch servers

So it is not related to HostHatch, and yet it is related to HostHatch?! Maybe I should wish you good luck in buying more (I guess), since you're already aware you'll need luck when you rely on your own backups for production.

I have redundant servers and providers, so if all HH servers fail, I have zero problems and everything will be running as usual from other providers, just with less local presence in some world regions.

If you depend only on one provider, being HH or other, you are going to have problems.

HH has good hardware and networking. If everything works as expected you have an excellent service for the price.

I accept some flaws and have low expectations. I'm paying $18 per server. What can I expect for this price?

FrankZ · March 2022

Email from HH today.

This is the third and final billing notice that your invoice no. xxxxx which was generated on 2022-03-23 is now overdue. Failure to make payment will result in account termination. Your server and data will be removed and unrecoverable in the case of termination.

We'll see what happens over the next few days, but I have moved my data to other non HH servers in anticipation that the eight servers in my account will be terminated and the data wiped. Still no response to the ticket. PITA.

skorous · March 2022

yeah, they say one thing and their automated process says something else. You can't take the risk just in case the automated process wins out.

root · March 2022

HostHatch was great in the past. I strongly feel it is no longer the case. I understand poor support during Black Friday because of overwhelming number of orders, but things like:

poor support now, when there is no overwhelming Black Friday,
downtime without transparency - Stor4-Chicago had RAID replaced in August 2021 (we had a downtime and maintenance for that).
forcing existing down services onto some beta panel, just to take advantage of a customer's frustrating problem in data loss;
creating new services which absolutely don't have the specs of the old servers, and then ask customers to move their data as if the new server is just like the old one, and like the old drive would fit the new one;
creating invoices which provider can't be bothered to cancel using some web script - instead charges the account automatically;
a new beta panel which is not transparent (neither in billing, nor in service specs), yet it is used for production.
a business plan to move all services in the beta panel, without customer having an option, and without at least informing the customer why they do it (in case provider is forced to do this).

All these problems have changed my mind; and I honestly loved HostHatch before.

Now I am in the process of copying data to other storages. I can only see HostHatch good as a third backup (backup of a second backup), and that's about it. I advise others to rethink and reassess this provider, because things are not what they used to be in HostHatch.

FrankZ · March 2022

@FrankZ said: I'll keep my comments short and calm here. I received a new email this morning that HH unsuccessfully tried to charge my c/c for the now overdue invoice of the new storage server. Still no response to the ticket.

The ticket was responded to today. All issues have been resolved.

TheDP · March 2022

That reminds me that I have a service transfer request that's been pending for almost 3 weeks now

Hosthatch - partial Chicago downtime

Comments