FlowVPS Message- Data array corruption

vyas

Received this mail from Flowvps @Brendan (thanks for the heads up, and the message underscores once again the importance of backups, and backups of backups! )

Hello,

If you are receiving this email, VPS services you have with us located on NODE5 have had their data unrecoverably lost. Recovery attempts have been made however the data array has become unrecoverably corrupt.

It is unfortunate this email needs to be sent however it is necessary you know as soon as possible to implement your disaster recovery plan.

Further updates will be provided about redeployment of services and 3 months of service credits will be added to all accounts effected.

Sorry for any inconvenience this may cause.

Regards,
Brendan

edit: corrected typos

Neoon

Well, I did not get that message but my monitoring says ded beef.

bdl

my microlxc and my other Flowvps VPS both down. Thank fully I got Proxmox backup server beta working (at another site) last night

Neoon

@bdl said:
my microlxc and my other Flowvps VPS both down. Thank fully I got Proxmox backup server beta working (at another site) last night

Yea I noticed, as soon I have confirmation, will see.

bdl

@Neoon said:

@bdl said:
my microlxc and my other Flowvps VPS both down. Thank fully I got Proxmox backup server beta working (at another site) last night

Yea I noticed, its DEAD, very bad.

It's probably all @Brendan needs with the covid-related lockdown in Melbourne - hope he has luck getting it all back on track, was very impressed with the proactive notice

I'm getting old, thinking I locked myself out of my microLXC due to borking a whitelist, but then realised it was hosted at FlowVPS need more beer

vyas

@SmallWeb Melbourne service seems to be okay- though I got multiple downtime alerts last night IST (GMT + 5.5)

AlwaysSkint

I'm glad I've got QuantumCore to fall back on, for my sole Oz client. My DR plan has a few wee niggles (email forwarders) but ain't too bad considering the e-commerce site is running under CWP. (The inbuilt CWP backups are crap!)

Time on Error: 3 hours 2 minutes 31 seconds

Used my tertiary backup (Online/Scaleway) to retrieve the files from other side of the globe, this time but I think I need to re-instate the Oz->Oz primary backup.

Haste ye back, @Brendan

(A special shoutout to @QuantumCore for being there when needed the most - previous time being HostDoc demise. I'm sure the same won't happen here! )

MichaelCee

@vyas said:
@SmallWeb Melbourne service seems to be okay- though I got multiple downtime alerts last night IST (GMT + 5.5)

Wouldn't know what node I am on, but no email received. Last remote backups Friday 21st.

Would be interested to know what's setting your monitor off as mine is fine:
https://status.smallweb.net/report/uptime/7ca1e97ad9b8383cb61fc2d43b60ecb5/

AlwaysSkint

Time to start again..

All VPS services are now ready to be reprovisioned.

MichaelCee

Very unfortunate. Best wishes to all.

seriesn

Ouch. That sucks ballz

Ympker

That sucks Best wishes to those affected! Hopefully things'll work out quickly

sonic

Hello,

Thank you for your patience while this is resolved.

All VPS services are now ready to be reprovisioned.

Please use the service control area to reinstall your VPS to gain access, this will give you a clean install and you will be issued with a root password. If you'd prefer to do a custom install via ISO that option is also available.

Please reach out via a support ticket if you need any help with the above task or if you have any other issues.

Over the next week any services affected by this will have their next due date pushed forward 3 months, no action is required on your side.

Regards,
Brendan

MonstaAU

My idling VPS is dead. Meh.

InceptionHosting

@MonstaAU said:
My idling VPS is dead. Meh.

Did you backup your idle?

you can restore with touch /nothing

vyas

Anybody run YABS or any other BM on the new and revived FPS from this node?
I got the following speeds (ticket of course being raised) but just checking reg disk speeds NVMe

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 79.96 MB/s   (19.9k) | 341.99 MB/s   (5.3k)
Write      | 80.17 MB/s   (20.0k) | 343.79 MB/s   (5.3k)
Total      | 160.13 MB/s  (40.0k) | 685.79 MB/s  (10.7k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 295.00 MB/s    (576) | 334.80 MB/s    (326)
Write      | 310.67 MB/s    (606) | 357.09 MB/s    (348)
Total      | 605.67 MB/s   (1.1k) | 691.90 MB/s    (674)

bdl

@bdl said:

@Neoon said:

@bdl said:
my microlxc and my other Flowvps VPS both down. Thank fully I got Proxmox backup server beta working (at another site) last night

Yea I noticed, its DEAD, very bad.

It's probably all @Brendan needs with the covid-related lockdown in Melbourne - hope he has luck getting it all back on track, was very impressed with the proactive notice

@vyas said:
Anybody run YABS or any other BM on the new and revived FPS from this node?
I got the following speeds (ticket of course being raised) but just checking reg disk speeds NVMe

> fio Disk Speed Tests (Mixed R/W 50/50):
> ---------------------------------
> Block Size | 4k            (IOPS) | 64k           (IOPS)
>   ------   | ---            ----  | ----           ---- 
> Read       | 79.96 MB/s   (19.9k) | 341.99 MB/s   (5.3k)
> Write      | 80.17 MB/s   (20.0k) | 343.79 MB/s   (5.3k)
> Total      | 160.13 MB/s  (40.0k) | 685.79 MB/s  (10.7k)
>            |                      |                     
> Block Size | 512k          (IOPS) | 1m            (IOPS)
>   ------   | ---            ----  | ----           ---- 
> Read       | 295.00 MB/s    (576) | 334.80 MB/s    (326)
> Write      | 310.67 MB/s    (606) | 357.09 MB/s    (348)
> Total      | 605.67 MB/s   (1.1k) | 691.90 MB/s    (674)
> 

here you go, just ran now

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 77.46 MB/s   (19.3k) | 406.85 MB/s   (6.3k)
Write      | 77.67 MB/s   (19.4k) | 408.99 MB/s   (6.3k)
Total      | 155.14 MB/s  (38.7k) | 815.85 MB/s  (12.7k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 337.10 MB/s    (658) | 412.93 MB/s    (403)
Write      | 355.02 MB/s    (693) | 440.43 MB/s    (430)
Total      | 692.12 MB/s   (1.3k) | 853.36 MB/s    (833)

vyas

@bdl said:

It's probably all @Brendan needs with the covid-related lockdown in Melbourne

here you go, just ran now

> fio Disk Speed Tests (Mixed R/W 50/50):
> ---------------------------------
> Block Size | 4k            (IOPS) | 64k           (IOPS)
>   ------   | ---            ----  | ----           ---- 
> Read       | 77.46 MB/s   (19.3k) | 406.85 MB/s   (6.3k)
> Write      | 77.67 MB/s   (19.4k) | 408.99 MB/s   (6.3k)
> Total      | 155.14 MB/s  (38.7k) | 815.85 MB/s  (12.7k)
>            |                      |                     
> Block Size | 512k          (IOPS) | 1m            (IOPS)
>   ------   | ---            ----  | ----           ---- 
> Read       | 337.10 MB/s    (658) | 412.93 MB/s    (403)
> Write      | 355.02 MB/s    (693) | 440.43 MB/s    (430)
> Total      | 692.12 MB/s   (1.3k) | 853.36 MB/s    (833)
> 

thanks, just learnt from colleagues at woork from AU that they are in Stage 4 lockdown. Maybe will hold off on the ticket for now and let it idle a bit more

Brendan

@vyas said:

@bdl said:

It's probably all @Brendan needs with the covid-related lockdown in Melbourne

here you go, just ran now

> > fio Disk Speed Tests (Mixed R/W 50/50):
> > ---------------------------------
> > Block Size | 4k            (IOPS) | 64k           (IOPS)
> >   ------   | ---            ----  | ----           ---- 
> > Read       | 77.46 MB/s   (19.3k) | 406.85 MB/s   (6.3k)
> > Write      | 77.67 MB/s   (19.4k) | 408.99 MB/s   (6.3k)
> > Total      | 155.14 MB/s  (38.7k) | 815.85 MB/s  (12.7k)
> >            |                      |                     
> > Block Size | 512k          (IOPS) | 1m            (IOPS)
> >   ------   | ---            ----  | ----           ---- 
> > Read       | 337.10 MB/s    (658) | 412.93 MB/s    (403)
> > Write      | 355.02 MB/s    (693) | 440.43 MB/s    (430)
> > Total      | 692.12 MB/s   (1.3k) | 853.36 MB/s    (833)
> > 

thanks, just learnt from colleagues at woork from AU that they are in Stage 4 lockdown. Maybe will hold off on the ticket for now and let it idle a bit more

Please raise one, I'll take a look. But lower than it should be.

Thanks everyone for being good about this, not an ideal situation for anyone.

vyas

@Brendan said:

Please raise one, I'll take a look. But lower than it should be.

Thanks everyone for being good about this, not an ideal situation for anyone.

I am okay to wait @Brendan, so that you can fight the bigger fires first. Will raise the ticket in due time.
Cheers