OVHcloud datacenter 'lacked' automatic fire extinguishers, electrical cutoff

serverollie

https://www.theregister.com/2022/03/22/ovhcloud_fire_datacenter_report/

No automatic fire system or electrical cut off and customers are taking a class actions because did not understand the need to pay for additional backups?

Over 140 customers have filed a class action lawsuit against the company seeking damages for losses arising from the fire. According to the law firm handling the complaint, Ziegler & Associates, numerous French government websites were affected by the fire, including data.gouv.fr, the National Education website, the Center Pompidou website and Meteosky. The law firm claims many customers lost their data as a result of the fire and did not understand the need to pay for additional backups.

You wonder how the rest of the OVH data centres are built based on this?

cybertech

will insurance company reject claims due to lack of fire suppression system?

lentro

So I was curious and searched up OVH's financials:

https://www.businesswire.com/news/home/20211115006403/en/OVHcloud-FY2021-financial-performance-at-the-high-end-of-the-guidance

398 million in revenue (euros) came from private cloud, whereas only 94 came from public cloud

--> A lot of high paying "enterprise" customers. Of their revenue, a majority comes from within France.

So, their largest source of revenue is French corporations or the French government, basically. Many of these might have been using SBG, or even if they were using RBX, they'd probably know about this situation quite well. OVH fucked up a little bit here; they didn't have to cheap out that much when so much of their customer base comprises of these rich French companies who would probably give more of a shit about losing data and less of a shit about paying more. Given how their EBITDA margins are 40%, they're probably regretting that they didn't lower their margins by literally 1% by installing fire suppression systems.

havoc

@cybertech said: will insurance company reject claims due to lack of fire suppression system?

The whole thing is just a bonfire of note

It took the local elec utility company THREE HOURS to get the power cut..

electrical arcs more than one meter long flashing around the door to the power room

Imagine this had been a flood instead of a fire. Would have been a real life half-life don't touch the floor game

Never mind fire...they better start with basic elec compliance

deank

Well, it was a bottom barrel DC. What did you expect?

Not too surprised.

vyas

@cybertech said:
will insurance company reject claims due to lack of fire suppression system?

You assume they had insurance :-).

Long answer

I asked the Mrs who heads risk management for a Corporate. Here are some of the points.
a. Was the DC in compliance with the building code of the land? Was the material used in construction - panels, cabling, etc. as per building code? If yes, then insurance company may move on to the following:
b. Fire suppression system - a bit of an enigma here in India. FM200 system puts out fire but fucks up the equipment. So what are you really trying to save- a shell of building or the capital equipment? Building code does not specify it, some companies with their global standards install it, and it is a pain in the rear end to clean up.
c. Were fire drills conducted, recorded?
d. Had the insurance company conducted their own risk assessment ? In a DC environment, risk coverage in decreasing order of magnitude
Data >> Data storage and processing devices >> building
e. Were the systems - capital equipment etc. as per ISO/ ABCDEFGH agency recommendations?
If you pick up the LEED (so called Green building) standards book, you will find atleast 130 different agencies and standards. Building code of India has over 1000. You can choose and pick, or complete a checklist without knowing your head from the ass.

Above is just a starting point. What about some unique quirks? Some companies like TCS (not sure if they still do) require placing fire extinguishers in front of offices. What was their safety officer smoking, the Almighty knows. But it effectively reduced the width of the corridor, wheelchairs require clear space on either side. That requirement was not met. etc...

Anyways, customers who sued need to get their head scans done. They opt for a cheap provider, and rely on provider's backup ??

---

Note: During my amazon days, there was a 45 or 60 second rule- there should be adequate exits in the warehouse to allow all people to exit the building within (45 or 60) seconds. Including those who need assistance. DC has far less people, so could be a different story.

cybertech

@vyas said:

@cybertech said:
will insurance company reject claims due to lack of fire suppression system?

You assume they had insurance :-) . I asked the Mrs who heads risk management for a Corporate.

Long answer

Here are some of the points.
a. Was the DC in compliance with the building code of the land? Was the material used in construction - panels, cabling, etc. as per building code? If yes, then insurance company may move on to the following:
b. Fire suppression system - a bit of an enigma here in India. FM200 system puts out fire but fucks up the equipment. So what are you really trying to save- a shell of building or the capital equipment? Building code does not specify it, some companies with their global standards install it, and it is a pain in the rear end to clean up.
c. Were fire drills conducted, recorded?
d. Had the insurance company conducted their own risk assessment ? In a DC environment, risk coverage in decreasing order of magnitude
Data >> Data storage and processing devices >> building
e. Were the systems - capital equipment etc. as per ISO/ ABCDEFGH agency recommendations?
If you pick up the LEED (so called Green building) standards book, you will find atleast 130 different agencies and standards. Building code of India has over 1000. You can choose and pick, or complete a checklist without knowing your head from the ass.

Above is just a starting point. What about some unique quirks? Some companies like TCS (not sure if they still do) require placing fire extinguishers in front of offices. What was their safety officer smoking, the Almighty knows. But it effectively reduced the width of the corridor, wheelchairs require clear space on either side. That requirement was not met. etc...

Anyways, customers who sued need to get their head scans done. They opt for a cheap provider, and rely on provider's backup ??

---

Note: During my amazon days, there was a 45 or 60 second rule- there should be adequate exits in the warehouse to allow all people to exit the building within (45 or 60) seconds. Including those who need assistance. DC has far less people, so could be a different story.

i think we now know who OVH India needs to pay top dollars for.

vyas

For those interested:

A good simulation of the FM200 system, btw.
Turn of volume, watch at 2x speed- because it is 4 min length, 2 minutes too long IMO.
Note the downside: if someone is inside the racks area, they would better exit ASAP.