VirMach - Complain - Moan - Praise - Chit Chat

VirMach

@cyforex said:
my whole service is down! I also can't open my service page in control panel, I don't know what node it is! the location is NYC.
my service ID: 691792
man your support system is a mess
@VirMach can you check?

If you don't know the node name, but know it's NYC, and there is currently around the same time a node down in NYC which is NYCB028, you can at that point probably safely assume it's likely that one.

Jab

Seems fun?

Update: unfortunately, the datacenter in Dallas has broken access to our out of bound management without notifying us previously, and then powered down the wrong server (DALZ005) causing a second outage during our maintenance request. We're still waiting for them to correct our access to DALZ004 so we can complete the reboot.

AuroraZero

@Jab said:
Seems fun?

Update: unfortunately, the datacenter in Dallas has broken access to our out of bound management without notifying us previously, and then powered down the wrong server (DALZ005) causing a second outage during our maintenance request. We're still waiting for them to correct our access to DALZ004 so we can complete the reboot.

When it rains, it pours it seems.

BHS

@VirMach said:

@cyforex said:
my whole service is down! I also can't open my service page in control panel, I don't know what node it is! the location is NYC.
my service ID: 691792
man your support system is a mess
@VirMach can you check?

If you don't know the node name, but know it's NYC, and there is currently around the same time a node down in NYC which is NYCB028, you can at that point probably safely assume it's likely that one.

It looks like NYCB022 has also dropped by this time.

JoeMerit

Every NYC server is down. Dedipath offline?

julensm

NYCB040 is down here
Operation Timed Out After 90000 Milliseconds With 0 Bytes Received

tenpera

Ny Nodes are all offline, Operation Timed Out After 90000 Milliseconds With 0 Bytes Received

Jab

I guess it's this
https://status.dedicated.com/

aka

The datacenter is experiencing a fire and is in emergency power off.

FrankZ

@JoeMerit said: Every NYC server is down. Dedipath offline?

The outage seems bigger than just VirMach. My VMs from another provider in the same DC when down at the same time as my VirMach VMs, and are still down.

AuroraZero

@Jab said:
I guess it's this
https://status.dedicated.com/

aka

The datacenter is experiencing a fire and is in emergency power off.

The other shoe definitely just dropped on this one. I hope everything gets sorted quickly with minimal damage to all parties involved, especially no one was, or is hurt.

nightcat

NYC server down.

Jab

@nightcat said:
NYC server down.

why read when you can just post.

AuroraZero

@Jab said:

@nightcat said:
NYC server down.

why read when you can just post.

Moar fun that way?

tenpera

@virmach, none of the services can connect to Solus, are you doing any?

localhost

@tenpera said:
@virmach, are you doing any?

Yes... he is doing all servers now

titus

@JoeMerit said:
Every NYC server is down. Dedipath offline?

My VPS from Dedipath directly (NY location) is also offline / unreachable.

This is from DediPath (because probably the same issue in the background):

loyaltyforge_dan

It looks like the latest is:

Power remains off at our data center in Secaucus/EWR1 per the local fire marshal.

We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process.

We are currently waiting on the fire marshal and local utility to reenergize the site. We are completely dependent upon their inspection and approval. We are hoping to get an update that we can share in the next hour.

At the current time, the fire department is controlling access to the building and we will not be able to let customers in.

JDMcPea

skorous

I blame @Virmach. He got his ticket queue down too low so now the DC had a fire to pump the queue back up.

Flying_Chinaman

@FrankZ said:

@cybertech said:

@Flying_Chinaman said:
Yo VirMach ,

1. What is the average HDD lifespan on your colocations? Will ya purchase cheap SMR to cut cost?

2. Which PCIe gen and brand Nvme SSD are ya using? How much IO it will fall after exceed its TBW?

also when will we colonize Mars?

And while your at it how can we resolve hunger in the Americas and establish world peace ?

O'it is my little guilty pleasure to nag VirmAch during defecting,

Ooo... just my tummy hurts recently

adef_1

Weekend has ended, and the NYCB027 node is still displaying as locked.

FrankZ

@Flying_Chinaman said: Ooo... just my tummy hurts recently

hanru

@Jab said:
I guess it's this
https://status.dedicated.com/

aka

The datacenter is experiencing a fire and is in emergency power off.

I feel guilty because I name my VPS in NYC fire.

Mumbly

Seems like NYCB028 pre-emptive shut itself down several hours before fire started

ZA_capetown

Life seems to be intermittedly returning to NYCB?

Got brief moments of seemingly working control panel (showing server as shut down).
But didn't manage to boot up yet.
Messages vary between timeout and
"There seems to have been a slight problem with our database, please try again later."

Hopefully SOON™

Flying_Chinaman

Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?

sh97

@Flying_Chinaman said:
Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?

5950, don't think 7000 series is deployed anywhere.

ZA_capetown

Update from the datacenter below. Their message did not specify a next update time, I've asked when we can expect to be updated.

Power remains off at our data center in Secaucus/EWR1 per the local fire marshal.

Current status update from DC Ops:

Our remediation vendor and our team has worked through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and we hope to have the cleaning completed by mid-day, at which time we will engage the fire marshal to review the site. Following their review, we hope to get a sign off from them so that we can start the reenergizing process. The reenergizing process can take 4-5 hours, as we need to turn up the critical infrastructure prior to any servers.

Posted 51 minutes ago by dedicated

untoreh

fire season is for forests datacenters

skorous

@Jab said:
Seems fun?

Update: unfortunately, the datacenter in Dallas has broken access to our out of bound management without notifying us previously, and then powered down the wrong server (DALZ005) causing a second outage during our maintenance request. We're still waiting for them to correct our access to DALZ004 so we can complete the reboot.

( chuckle ) Interesting ... what would constitute "out of bound management"?