Hosthatch down?

bjobjo OG
edited September 2023 in Outages

I wanted to reboot a Stockholm VM into an ISO. "Booting up" -> "Offline". Wanted to create a ticket. Now Cloudflare tells me cloud.hosthatch.com is down.

Tagged:
«1

Comments

  • Isn't console the old panel that was supposed to be dead?
    https://cloud.hosthatch.com/login

    My server in Amsterdam is online.

    Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
    https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png

  • cloud.hosthatch.com is up and so is my VPS (Netherlands)

  • Both cloud.hosthatch.com and my VPS in Stockholm are accessible.

  • At leat the console is up again, so I can create a ticket regarding the boot issue.

  • My storage server in NL is down since 15:00 GMT.
    Waiting for their response to my ticket.

  • edited September 2023

    Storage server in Stockholm: pings over ipv4/v6, but otherwise unconnectable/unresponsive. The online console reveals a ton of error messages over the last few hours. Reboot/poweroff through the website fails. Host/hardware crash?

    YAY DRAMA, BILLIONS LOST

    edit: things are back

  • HH storage STO Confirmed downtime 18min according to hetrix, now back again

  • Hetrixtools show 8 minutes downtime.

  • Seems to be an ipv6 issue again.

  • I don't monitor IPv6 uptime here (my bad, I should...) but right this moment when I saw your post and checked, IPv6 works for me.

  • Support replied they had a hardware failure in Stockholm.

  • Same here the StockHolm VM was unbootable from the panel yesterday. Seems to be fixed now but IPv6 is now broken.

  • I am wondering if we're on the different nodes. No outage here and IPv6 is up & running. Last time was right the opposite. IPv6 was up for everyone except for me.

    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2023-09-06                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Wed 20 Sep 10:44:00 CEST 2023
    
    Basic System Information:
    ---------------------------------
    Uptime     : 14 days, 21 hours, 45 minutes
    Processor  : AMD EPYC 7443P 24-Core Processor
    CPU cores  : 2 @ 2844.656 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 7.8 GiB
    Swap       : 975.0 MiB
    Disk       : 53.8 GiB
    Distro     : Debian GNU/Linux 12 (bookworm)
    Kernel     : 6.1.0-11-amd64
    VM Type    : KVM
    IPv4/IPv6  : ✔ Online / ✔ Online
    
    IPv6 Network Information:
    ---------------------------------
    ISP        : HostHatch
    ASN        : AS63473 HostHatch, LLC
    Host       : HostHatch, LLC
    Location   : Stockholm, Stockholm County (AB)
    Country    : Sweden
    
    fio Disk Speed Tests (Mixed R/W 50/50):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 278.28 MB/s  (69.5k) | 3.34 GB/s    (52.2k)
    Write      | 279.02 MB/s  (69.7k) | 3.35 GB/s    (52.4k)
    Total      | 557.30 MB/s (139.3k) | 6.70 GB/s   (104.6k)
               |                      |                     
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 4.67 GB/s     (9.1k) | 4.89 GB/s     (4.7k)
    Write      | 4.92 GB/s     (9.6k) | 5.21 GB/s     (5.0k)
    Total      | 9.59 GB/s    (18.7k) | 10.10 GB/s    (9.8k)
    
    iperf3 Network Speed Tests (IPv4):
    ---------------------------------
    Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
    -----           | -----                     | ----            | ----            | ----           
    Clouvider       | London, UK (10G)          | 153 Mbits/sec   | 359 Mbits/sec   | 26.4 ms        
    Scaleway        | Paris, FR (10G)           | busy            | busy            | 38.6 ms        
    NovoServe       | North Holland, NL (40G)   | busy            | busy            | 20.7 ms        
    Uztelecom       | Tashkent, UZ (10G)        | busy            | 1.28 Gbits/sec  | 112 ms         
    Clouvider       | NYC, NY, US (10G)         | 41.7 Mbits/sec  | 59.8 Mbits/sec  | 96.2 ms        
    Clouvider       | Dallas, TX, US (10G)      | 13.7 Mbits/sec  | 77.0 Mbits/sec  | 124 ms         
    Clouvider       | Los Angeles, CA, US (10G) | 25.2 Mbits/sec  | 35.8 Mbits/sec  | 153 ms         
    
    iperf3 Network Speed Tests (IPv6):
    ---------------------------------
    Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
    -----           | -----                     | ----            | ----            | ----           
    Clouvider       | London, UK (10G)          | 153 Mbits/sec   | 382 Mbits/sec   | 26.2 ms        
    Scaleway        | Paris, FR (10G)           | 5.03 Gbits/sec  | 3.11 Gbits/sec  | 35.5 ms        
    NovoServe       | North Holland, NL (40G)   | busy            | busy            | 20.8 ms        
    Uztelecom       | Tashkent, UZ (10G)        | busy            | 1.20 Gbits/sec  | 112 ms         
    Clouvider       | NYC, NY, US (10G)         | 40.8 Mbits/sec  | 84.8 Mbits/sec  | 96.1 ms        
    Clouvider       | Dallas, TX, US (10G)      | 31.0 Mbits/sec  | 77.2 Mbits/sec  | 125 ms         
    Clouvider       | Los Angeles, CA, US (10G) | 25.4 Mbits/sec  | 41.7 Mbits/sec  | 153 ms         
    
    Geekbench 6 Benchmark Test:
    ---------------------------------
    Test            | Value                         
                    |                               
    Single Core     | 1824                          
    Multi Core      | 3339                          
    
  • @Mumbly said: I am wondering if we're on the different nodes. No outage here and IPv6 is up & running.

    Could have been related to the storage servers.

    My E5 NVMe server has been up.

    Thanked by (1)Mumbly
  • @TheDP said: Could have been related to the storage servers.

    Ah, this would explain it. I don't have storage server there.

  • Indeed mine is a storage server in Stockholm.

    Thanked by (1)Mumbly
  • @beagle said:
    My storage server in NL is down since 15:00 GMT.
    Waiting for their response to my ticket.

    Their response:

    The host node your VM is being hosted on suffered from a hardware failure and we had to dispatch a technician to replace the RAID controller. It is now back online and we are continue to monitoring it closely. We are sorry for the inconvenience caused.

  • Storage server in SE back up, but IPv6 is now broken :p

    Thanked by (1)bjo
  • Same here:

    PING fe80::1(fe80::1) 56 data bytes
    ^C
    --- fe80::1 ping statistics ---
    2 packets transmitted, 0 received, 100% packet loss, time 1024ms
    
  • Update from HH LA storage server:

    Dear Customer,

    This is an important message regarding your server(s);
    server (x.x.x.x) - Los Angeles

    Due to a multi disk failure in the RAID50 array, we were not able to restore your Storage VM (mentioned above) to its previous state. Please login to your account at cloud.hosthatch.com and use the reinstall feature to start using it again.

    We have moved your server to a node with RAID10 HDD storage at no additional cost, for better redundancy and performance of your VM in the future. We will also be issuing 30 days of credit to your account later this week.

    Thank you for your understanding and we apologize for the inconvenience caused.

  • cybertechcybertech OGBenchmark King
    edited September 2023

    @beagle said:
    Update from HH LA storage server:

    Dear Customer,

    This is an important message regarding your server(s);
    server (x.x.x.x) - Los Angeles

    Due to a multi disk failure in the RAID50 array, we were not able to restore your Storage VM (mentioned above) to its previous state. Please login to your account at cloud.hosthatch.com and use the reinstall feature to start using it again.

    We have moved your server to a node with RAID10 HDD storage at no additional cost, for better redundancy and performance of your VM in the future. We will also be issuing 30 days of credit to your account later this week.

    Thank you for your understanding and we apologize for the inconvenience caused.

    RAID10 ftw and they should have done this from the get go.

    I bench YABS 24/7/365 unless it's a leap year.

  • Seems like bandwidth accounting does not work any more. Stockholm storage is stuck at 300mb and Amsterdam has not used anything yet.

  • edited September 2023

    @bjo said: Seems like bandwidth accounting does not work any more. Stockholm storage is stuck at 300mb and Amsterdam has not used anything yet.

    It sounds like 80% of my VPSes :P

  • @Mumbly said:
    It sounds like 80% of my VPSes :P

    But I moved around 2 TB :D

  • IPv6 not working on the LA replacement server.
    Opened a ticket on 27th and I'm still waiting for a response.

  • @beagle said:
    IPv6 not working on the LA replacement server.
    Opened a ticket on 27th and I'm still waiting for a response.

    same too, that's too bad

  • Looks like my Stockholm VM is off again (same problem as reported in this thread).
    Reboots fail and console just 'Waiting' - panel shows as @ offline.
    Power off and boot, panel showing VM offline
    Also getting cloudflare timeouts when accessing https://cloud.hosthatch.com/

  • Storage or NVMe package?
    My NVMe VPS is up.

  • My Stockholm storage vm is also down.

  • bjobjo OG
    edited October 2023

    Same here with storage, but seems up again.

Sign In or Register to comment.