@JoeMerit said: Every NYC server is down. Dedipath offline?
The outage seems bigger than just VirMach. My VMs from another provider in the same DC when down at the same time as my VirMach VMs, and are still down.
Same for me. This is by far the longest downtime I've had with HawkHost since 2009.
Those, who look for another source of updates on New York Secaucus data center fire, can check https://hawkhoststatus.com/
☰ Probably the best Black Friday storage offers — AMD EPYC VDSes with NVMe slices (ref) from 250GB to 4TB and 500GB–10TB SAN disk. ／ Big HDD storage VPSes (ref) from $2.42/month per TB. ／ Storage dedis and hybrid VPS (SSD + HDD) are there as well.
@cyforex said:
my whole service is down! I also can't open my service page in control panel, I don't know what node it is! the location is NYC.
my service ID: 691792
man your support system is a mess @VirMach can you check?
If you don't know the node name, but know it's NYC, and there is currently around the same time a node down in NYC which is NYCB028, you can at that point probably safely assume it's likely that one.
It looks like NYCB022 has also dropped by this time.
NYCB028 update, our emergency power cycle request was taken way too seriously when I told them to do whatever it takes.
@Flying_Chinaman said:
Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?
5950, don't think 7000 series is deployed anywhere.
I haven't checked since a few months ago but at that time, 7900X just seemed like a very not good idea. Higher thermals, which you could scale back to lose performance, and you can't do faster memory if you want to do 128GB and have to scale back to 64GB or run at similar speeds as DDR4 except with added latency.
Only thing that tempted me was the pins no longer being on the CPU which means we could do a successful CPU reseating DC hands request without damaging the CPU.
(edit) Oh and Gen5 NVMe was pretty much useless around the time I checked but I'd imagine that's changed.
@VirMach said: Only thing that tempted me was the pins no longer being on the CPU which means we could do a successful CPU reseating DC hands request without damaging the CPU.
random tech working for [censored]: Challenge accepted, hold my beer.
@Flying_Chinaman said:
Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?
5950, don't think 7000 series is deployed anywhere.
I haven't checked since a few months ago but at that time, 7900X just seemed like a very not good idea. Higher thermals, which you could scale back to lose performance, and you can't do faster memory if you want to do 128GB and have to scale back to 64GB or run at similar speeds as DDR4 except with added latency.
Only thing that tempted me was the pins no longer being on the CPU which means we could do a successful CPU reseating DC hands request without damaging the CPU.
(edit) Oh and Gen5 NVMe was pretty much useless around the time I checked but I'd imagine that's changed.
why is there a need to reseat CPU if the CPU is well seated before?
@Flying_Chinaman said:
Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?
5950, don't think 7000 series is deployed anywhere.
I haven't checked since a few months ago but at that time, 7900X just seemed like a very not good idea. Higher thermals, which you could scale back to lose performance, and you can't do faster memory if you want to do 128GB and have to scale back to 64GB or run at similar speeds as DDR4 except with added latency.
The 7900 non-X varient isn't a bad processor, only 65W TDP and super power efficient while still having great performance.
@VirMach said: Only thing that tempted me was the pins no longer being on the CPU which means we could do a successful CPU reseating DC hands request without damaging the CPU.
random tech working for [censored]: Challenge accepted, hold my beer.
True story.
We had a server blade with motherboard but no CPU/RAM.
Boss wanted me to order CPU and RAM so that it can be used again.
I determined the proper models that match the sockets, and submitted the request.
The order came in after several months.
A mate and I plugged in the CPU and RAM, planning to rack the server.
Boss walked in at that very moment, and asked: where are the heat sinks?
I didn't order any, because I had no idea that there need to be heat sinks.
I usually deal with Raspberry, which doesn't have heat sink.
If the boss didn't walk in at that time, the server would have went up in flames.
Later we found some heat sinks in the corner cabinet and installed them.
The server ran happily thereafter.
@VirMach said: Only thing that tempted me was the pins no longer being on the CPU which means we could do a successful CPU reseating DC hands request without damaging the CPU.
random tech working for [censored]: Challenge accepted, hold my beer.
Virmach’s nuts, yo' he never done this to be mean.
@cybertech said: why is there a need to reseat CPU if the CPU is well seated before?
The motherboard latch design is pretty bad, IMO.
Epyc has like a solid latch with triple screws and a slider mechanism since the need it since the CPU is so large. Ryzen 7000 and Intel have a pretty solid latch since they don't have pins on the CPU, but I guess when they have the pins on the CPU they just do a simple latch since it should theoretically be fine.
If you do a good job on the thermal paste and it makes a good seal, it grips onto the heatsink, then the latch allows it to slip out a little bit, enough to cause problems. Seems to happen more the further away the servers are shipped. Tokyo & LAX are okay, NYC not so much. All of this is just me guesstimating a lot of things, I don't have the actual numbers and so much has happened over the last year that it gets pushed out of my brain pretty quickly.
Like the latch will basically allow the CPU to come out with the heatsink even when it's properly closed, which isn't supposed to happen. And the four screws on the motherboard, even when tightened properly, might have just enough of a difference in tolerance to where it ends up slightly uneven as a result. Over time this can end up causing problems, in extreme cases it can kill the CPU especially if after a reseat request it gets damaged further instead of improved (and if it's damaged no tech is going to tell us they damaged it of course... so we won't know until we get it back.) Cases where it needs to definitely be reseated is for example when the server arrives and doesn't post.
To resolve this issue, at some point I made some modifications to the thermal paste to make it not grip so well. Unfortunately this ended up meaning that a few of them ran into thermal issues after some time, and thermal paste had to be re-done, and in some of those cases the CPU still came off because you need to still know how to take them off properly. For example if you don't properly rotate in between screws when opening it, if it's too cold, or if you pull it a certain way versus another it might still happen. So in those cases it still ended up being problematic. In some of these cases it seems like the tech got thermal paste underneath the CPU, which isn't good. At least one that came back that had fully died was a result of that.
@Flying_Chinaman said:
Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?
5950, don't think 7000 series is deployed anywhere.
I haven't checked since a few months ago but at that time, 7900X just seemed like a very not good idea. Higher thermals, which you could scale back to lose performance, and you can't do faster memory if you want to do 128GB and have to scale back to 64GB or run at similar speeds as DDR4 except with added latency.
The 7900 non-X varient isn't a bad processor, only 65W TDP and super power efficient while still having great performance.
Yep, that's what we would go with personally if we did ever do them. I guess I'd really want it to be top of the line if we made the jump, but it just didn't seem feasible. Like it'd be nice to say "yeah, we're going to do 7950X, put beefy cooling on it, and 5200MHz DDR5 with Gen5 NVMe." Instead of doing 3200MHZ to 3600MHz, keeping it Gen4, and the maybe getting 20% CPU performance increase with the 7900 (IIRC) when for most nodes it's not even the bottleneck.
@yoursunny said: If the boss didn't walk in at that time, the server would have went up in flames.
It's really difficult to melt newer CPUs unless you have some very custom settings going for them. Believe me, I've tried.
(edit) New upcoming special, additional 2% discount for VPS on a server with no heatsink. Exclusively collocated at Evocative NYJ. Delidded.
My personal highly speculative guess is that those updates are going to continue providing false hope. It seems to be how they're deciding to forecast things. They missed their first deadline, it went from night to morning to evening, and then either more work got stacked on top or they tried avoiding it and fire marshal wouldn't budge. Thursday or Friday if we're lucky and they've been truthful with their story, and I wouldn't be surprised if it goes into next week. Theoretically the earliest it could be is Wednesday evening. I don't know if I "trust" their initial story on what happened, I don't think it'd be as much of a problem as it's being right now for the fire marshal if it was just "some smoke" from one thing somewhere sectioned off from the rest of the facility.
Sounds like there's some definitive underlying electrical issues that caused it in the first place, and on top of that it sounds like their fire safety systems may have also failed if the fire department had to be called in to bring it under control.
@VirMach said: Theoretically the earliest it could be is Wednesday evening.
This may be possible now. I'm hesitant to update the network status page, I'll wait until they get an official go-ahead, right now it's just preliminary inspection that seems OK.
UPDATE - 1052 EST - 12th July:
I heard the preliminary inspection is good and we are taking steps to energize the property now.
I’m waiting for the official update from DC Ops. More to come.
(edit) Remember "energizing" will take like 4-5 hours. Then servers get powered on, however long that takes. And that's after official update. Sounds like it'll be in the evening (NY time) if it does get approved.
@VirMach said: (edit) Remember "energizing" will take like 4-5 hours. Then servers get powered on, however long that takes. And that's after official update. Sounds like it'll be in the evening (NY time) if it does get approved.
Energizing the site is preliminary to doing a full fire suppression system test. Considering how it's gone to this point there's at least a fair chance they fail at least some aspect of that.
"In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced."
@VirMach said:
My personal highly speculative guess is that those updates are going to continue providing false hope. It seems to be how
Actually ,we need news from the virmach website announcement.
I have planned to migrate site after some hours.
You should've done that significantly before now. There was a fire. Recoveries from those are never fast.
My favorite ticket today was someone who asked me to just move him to another server in NYC because he was tired of waiting so long. He said enough to indicate that he had definitely read the network status page and it was just interesting seeing him propose a rational solution based on what he had read.
@VirMach said: (edit) Remember "energizing" will take like 4-5 hours. Then servers get powered on, however long that takes. And that's after official update. Sounds like it'll be in the evening (NY time) if it does get approved.
Energizing the site is preliminary to doing a full fire suppression system test. Considering how it's gone to this point there's at least a fair chance they fail at least some aspect of that.
"In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced."
Yeah I'm still very cynical, just laying out what I consider to be the conservative minimums because the way they're doing updates makes it sound like they're already at the finish line.
Comments
Latest updates from DediPath:
https://i.imgur.com/V25a2xA.png
There we go, that makes more sense now. We were really missing a major setback and everything was looking suspiciously great this week.
Same for me. This is by far the longest downtime I've had with HawkHost since 2009.
Those, who look for another source of updates on New York Secaucus data center fire, can check https://hawkhoststatus.com/
☰ Probably the best Black Friday storage offers — AMD EPYC VDSes with NVMe slices (ref) from 250GB to 4TB and 500GB–10TB SAN disk. ／ Big HDD storage VPSes (ref) from $2.42/month per TB. ／ Storage dedis and hybrid VPS (SSD + HDD) are there as well.
O'server goes dung,
tickets are piling up,
precious Tokyo remain unseen
Chinaman still stay in China
how dare ya say it is great
Ontario Dildo Inspector
NYCB028 update, our emergency power cycle request was taken way too seriously when I told them to do whatever it takes.
titties
@Virmach how about shipping those servers to LA and racking them there? sound good?
O'my friend in tits, welcome to the masochist house
Yo' @VirMach when will ya launch a new thread instead of hiding behind proxy agent Franz
Ontario Dildo Inspector
Sounds like no one can even go into the building yet. Once there's a fire they act as if the entire place could spontaneously combust at any moment.
What, you've never played baseball with servers?
I haven't checked since a few months ago but at that time, 7900X just seemed like a very not good idea. Higher thermals, which you could scale back to lose performance, and you can't do faster memory if you want to do 128GB and have to scale back to 64GB or run at similar speeds as DDR4 except with added latency.
Only thing that tempted me was the pins no longer being on the CPU which means we could do a successful CPU reseating DC hands request without damaging the CPU.
(edit) Oh and Gen5 NVMe was pretty much useless around the time I checked but I'd imagine that's changed.
random tech working for [censored]: Challenge accepted, hold my beer.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
why is there a need to reseat CPU if the CPU is well seated before?
I bench YABS 24/7/365 unless it's a leap year.
The 7900 non-X varient isn't a bad processor, only 65W TDP and super power efficient while still having great performance.
I am a representative of Advin Servers
True story.
We had a server blade with motherboard but no CPU/RAM.
Boss wanted me to order CPU and RAM so that it can be used again.
I determined the proper models that match the sockets, and submitted the request.
The order came in after several months.
A mate and I plugged in the CPU and RAM, planning to rack the server.
Boss walked in at that very moment, and asked: where are the heat sinks?
I didn't order any, because I had no idea that there need to be heat sinks.
I usually deal with Raspberry, which doesn't have heat sink.
If the boss didn't walk in at that time, the server would have went up in flames.
Later we found some heat sinks in the corner cabinet and installed them.
The server ran happily thereafter.
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
Virmach’s nuts, yo' he never done this to be mean.
Ontario Dildo Inspector
The motherboard latch design is pretty bad, IMO.
Epyc has like a solid latch with triple screws and a slider mechanism since the need it since the CPU is so large. Ryzen 7000 and Intel have a pretty solid latch since they don't have pins on the CPU, but I guess when they have the pins on the CPU they just do a simple latch since it should theoretically be fine.
If you do a good job on the thermal paste and it makes a good seal, it grips onto the heatsink, then the latch allows it to slip out a little bit, enough to cause problems. Seems to happen more the further away the servers are shipped. Tokyo & LAX are okay, NYC not so much. All of this is just me guesstimating a lot of things, I don't have the actual numbers and so much has happened over the last year that it gets pushed out of my brain pretty quickly.
Like the latch will basically allow the CPU to come out with the heatsink even when it's properly closed, which isn't supposed to happen. And the four screws on the motherboard, even when tightened properly, might have just enough of a difference in tolerance to where it ends up slightly uneven as a result. Over time this can end up causing problems, in extreme cases it can kill the CPU especially if after a reseat request it gets damaged further instead of improved (and if it's damaged no tech is going to tell us they damaged it of course... so we won't know until we get it back.) Cases where it needs to definitely be reseated is for example when the server arrives and doesn't post.
To resolve this issue, at some point I made some modifications to the thermal paste to make it not grip so well. Unfortunately this ended up meaning that a few of them ran into thermal issues after some time, and thermal paste had to be re-done, and in some of those cases the CPU still came off because you need to still know how to take them off properly. For example if you don't properly rotate in between screws when opening it, if it's too cold, or if you pull it a certain way versus another it might still happen. So in those cases it still ended up being problematic. In some of these cases it seems like the tech got thermal paste underneath the CPU, which isn't good. At least one that came back that had fully died was a result of that.
Yep, that's what we would go with personally if we did ever do them. I guess I'd really want it to be top of the line if we made the jump, but it just didn't seem feasible. Like it'd be nice to say "yeah, we're going to do 7950X, put beefy cooling on it, and 5200MHz DDR5 with Gen5 NVMe." Instead of doing 3200MHZ to 3600MHz, keeping it Gen4, and the maybe getting 20% CPU performance increase with the 7900 (IIRC) when for most nodes it's not even the bottleneck.
It's really difficult to melt newer CPUs unless you have some very custom settings going for them. Believe me, I've tried.
(edit) New upcoming special, additional 2% discount for VPS on a server with no heatsink. Exclusively collocated at Evocative NYJ. Delidded.
Wha', @VirMach , if there won' be no Tokyo storage, will ya sen' more normal nodes?
Yo' poor Chinaman's longing for more dirty cheap Tokyo
B650 or X670? At leas' memory cap's increase'
I'm still on R7 2700x, currently considering skip entire AM5 platform and grab 5700x or 5800x3d
Ontario Dildo Inspector
Hi ,all
Any news about NY server?
Why read when you can just post, again.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Im wondering that too, i have 2 nodes, both still down... Is it possible to give a estimative on how it will still takes?
It's out of VirMach's hands right now. You can read this for progress on the datacenter: https://status.dedicated.com/incidents/44
My personal highly speculative guess is that those updates are going to continue providing false hope. It seems to be how they're deciding to forecast things. They missed their first deadline, it went from night to morning to evening, and then either more work got stacked on top or they tried avoiding it and fire marshal wouldn't budge. Thursday or Friday if we're lucky and they've been truthful with their story, and I wouldn't be surprised if it goes into next week. Theoretically the earliest it could be is Wednesday evening. I don't know if I "trust" their initial story on what happened, I don't think it'd be as much of a problem as it's being right now for the fire marshal if it was just "some smoke" from one thing somewhere sectioned off from the rest of the facility.
Sounds like there's some definitive underlying electrical issues that caused it in the first place, and on top of that it sounds like their fire safety systems may have also failed if the fire department had to be called in to bring it under control.
Super unofficial post, not an update.
Actually ,we need news from the virmach website announcement.
I have planned to migrate site after some hours.
Looks like there was just an update.
This may be possible now. I'm hesitant to update the network status page, I'll wait until they get an official go-ahead, right now it's just preliminary inspection that seems OK.
(edit) Remember "energizing" will take like 4-5 hours. Then servers get powered on, however long that takes. And that's after official update. Sounds like it'll be in the evening (NY time) if it does get approved.
You should've done that significantly before now. There was a fire. Recoveries from those are never fast.
Energizing the site is preliminary to doing a full fire suppression system test. Considering how it's gone to this point there's at least a fair chance they fail at least some aspect of that.
"In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced."
My favorite ticket today was someone who asked me to just move him to another server in NYC because he was tired of waiting so long. He said enough to indicate that he had definitely read the network status page and it was just interesting seeing him propose a rational solution based on what he had read.
Yeah I'm still very cynical, just laying out what I consider to be the conservative minimums because the way they're doing updates makes it sound like they're already at the finish line.
Latest updates from DediPath: https://i.imgur.com/61dOTbV.png
Hey, it's not all bad news. My VPS in Dallas is back up!