We have completed the full site inspection with the fire marshal and the electrical inspector and utility power has been restored to the site.
We are now working to restore critical systems and our onsite team has energized the primary electrical equipment that powers the site. Concurrently, we are beginning work to bring the mechanical plant online. Additional engineers from other facilities are on site this morning to expedite site turn up.
The ETA for bringing up the critical infrastructure systems is approximately 5 hours.
We are planning for a late afternoon/early evening time frame when clients will be able to come back on site.
We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
@VirMach said:
My favorite ticket today was someone who asked me to just move him to another server in NYC because he was tired of waiting so long. He said enough to indicate that he had definitely read the network status page and it was just interesting seeing him propose a rational solution based on what he had read.
Mentally strong VPS provider immediately leases another hypervisor in another nearby datacenter and migrates affected customers without data.
Our onsite team has energized the primary electrical equipment that powers the site, enabling us to bring our mechanical plant online. We are currently cooling the facility.
As we monitor for stability, we are focused on bringing up our electrical systems. In starting this process, we have identified an issue with powering up our fire panel as well as power systems that were powered by UPS3. While this will cause us a delay, we are working with our vendors for remediation.
We are currently at 25% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 7 hours. We are still planning for an evening time frame when clients will be able to come back on site. We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
While I am never happy to hear when an entire data center is offline, I am a bit relieved because I had 6 or 7 VPS go offline on Monday and users in this thread were only reporting that a single node was offline. I thought, oh man, how do I have 7 VPS on the same node? Turns out I don't.
Also things I learned since Monday: my old Chicago VPS specials are now located in NYC for some reason. shrug
Sounds like there's some definitive underlying electrical issues that caused it in the first place, and on top of that it sounds like their fire safety systems may have also failed if the fire department had to be called in to bring it under control.
Due to the amount of electricity flowing through these buildings, the fire department has to be involved every time, even if suppression works properly.
@yoursunny said: Mentally strong VPS provider immediately leases another hypervisor in another nearby datacenter and migrates affected customers without data.
Meanwhile mentally nuts VPS provider reluctantly updates the Network status page and mark related affected customers' ticket closed
Sounds like there's some definitive underlying electrical issues that caused it in the first place, and on top of that it sounds like their fire safety systems may have also failed if the fire department had to be called in to bring it under control.
Due to the amount of electricity flowing through these buildings, the fire department has to be involved every time, even if suppression works properly.
@tenpera said:
One of the N.Y node, NYCB035 is online.
I was monitoring everything all day waiting for them to return and go through to verify everything but then I spent the last hour accidentally coming across this.
Anyway, doing that now as planned. I specifically stayed up so I wouldn't miss it, going to go through all the nodes and validate everything's okay since things may be funky after a reboot. Might need another reboot for some of them just to ensure everything's good since some of these haven't rebooted in a while. I do see some of them are already running into issues so going over those first.
@VirMach said: I spent the last hour accidentally coming across this.
LEB strikes again with the "Expect VirMach to deadpool any minute now" saga. I find it odd that LEB is still writing about you. Were you once married to someone there ? Because the way they write about you is like an ex-wife/lover who is really pissed off. Tell the truth, did you and Jon have a thing going on the side ?
Looks like all of my VPS from VirMach is online again, which were affected in the NY datacenter outage.
(My VPS's from DediPath (directly) & from Ethernetservers are also online - they use the same DC).
@BHS said:
Several NYC are already active except NYCB028 is still down. Is it possible to reboot again, because NYCB028 was down before the fire occurred.
It does not seems like simple reboot will solve NYCB028 issue as everything in NYC was shut down anyway.
@tomsm said:
ny 022 is not work Offline PhysicalValuable-VM id：688689
Click on also can not be normal
This is the #1 issue and #1 reason we get tickets recently. People have an old ISO mounted, probably forget about it, after it gets rebooted a long time later, especially if we updated the ISO since then, SolusVM will not unmount it automatically and instead completely refuses to boot the VM back online without providing proper error on clientside.
I made a knowledgebase article for it already but all you need to know is you have to unmount ISO and then boot.
@Flying_Chinaman said: Also new control panel UI is rather inconvenient,
the serf who modified that should be hanged
All me, I did it after watching a few customers explain how they understood things. It's specifically tailored to reduce unnecessary support tickets but of course as with all changes it's had the opposite effect for the time being.
Biggest confusing thing for people I guess was hiding the password field, too many people relied on it I suppose which isn't good since you're supposed to grab it and then change it for security reasons. I'll see about adding that back in under the reset password section where it shows it. Any other complaints you have about it let me know.
@tomsm said:
ny 022 is not work Offline PhysicalValuable-VM id：688689
Click on also can not be normal
This is the #1 issue and #1 reason we get tickets recently. People have an old ISO mounted, probably forget about it, after it gets rebooted a long time later, especially if we updated the ISO since then, SolusVM will not unmount it automatically and instead completely refuses to boot the VM back online without providing proper error on clientside.
I made a knowledgebase article for it already but all you need to know is you have to unmount ISO and then boot.
Do you mean to reinstall the system? oh god my data
@tomsm said:
ny 022 is not work Offline PhysicalValuable-VM id：688689
Click on also can not be normal
This is the #1 issue and #1 reason we get tickets recently. People have an old ISO mounted, probably forget about it, after it gets rebooted a long time later, especially if we updated the ISO since then, SolusVM will not unmount it automatically and instead completely refuses to boot the VM back online without providing proper error on clientside.
I made a knowledgebase article for it already but all you need to know is you have to unmount ISO and then boot.
Do you mean to reinstall the system? oh god my data
I have no ISO mounted , after the power failure in NY it just don't want to boot anymore from whatever device HD/CDROM
If i mount a ISO it even crash and shutdown the instance too, VNC on or off all does not matter.
@tomsm said:
ny 022 is not work Offline PhysicalValuable-VM id：688689
Click on also can not be normal
This is the #1 issue and #1 reason we get tickets recently. People have an old ISO mounted, probably forget about it, after it gets rebooted a long time later, especially if we updated the ISO since then, SolusVM will not unmount it automatically and instead completely refuses to boot the VM back online without providing proper error on clientside.
I made a knowledgebase article for it already but all you need to know is you have to unmount ISO and then boot.
Do you mean to reinstall the system? oh god my data
What? No.
Can you assist me with the operation in the background?
@hike said:
I have no ISO mounted , after the power failure in NY it just don't want to boot anymore from whatever device HD/CDROM
If i mount a ISO it even crash and shutdown the instance too, VNC on or off all does not matter.
Reboot in 15 minutes, I'm taking a look at NYCB014 now and fixing any problems.
@hike said:
I have no ISO mounted , after the power failure in NY it just don't want to boot anymore from whatever device HD/CDROM
If i mount a ISO it even crash and shutdown the instance too, VNC on or off all does not matter.
Reboot in 15 minutes, I'm taking a look at NYCB014 now and fixing any problems.
Comments
Dear Valued Customers,
We have completed the full site inspection with the fire marshal and the electrical inspector and utility power has been restored to the site.
We are now working to restore critical systems and our onsite team has energized the primary electrical equipment that powers the site. Concurrently, we are beginning work to bring the mechanical plant online. Additional engineers from other facilities are on site this morning to expedite site turn up.
The ETA for bringing up the critical infrastructure systems is approximately 5 hours.
We are planning for a late afternoon/early evening time frame when clients will be able to come back on site.
We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Mentally strong VPS provider immediately leases another hypervisor in another nearby datacenter and migrates affected customers without data.
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
Fucking VirMach was right, again.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
While I am never happy to hear when an entire data center is offline, I am a bit relieved because I had 6 or 7 VPS go offline on Monday and users in this thread were only reporting that a single node was offline. I thought, oh man, how do I have 7 VPS on the same node? Turns out I don't.
Also things I learned since Monday: my old Chicago VPS specials are now located in NYC for some reason. shrug
Due to the amount of electricity flowing through these buildings, the fire department has to be involved every time, even if suppression works properly.
One of the N.Y node, NYCB035 is online.
Several NYC are already active except NYCB028 is still down. Is it possible to reboot again, because NYCB028 was down before the fire occurred.
Meanwhile mentally nuts VPS provider reluctantly updates the Network status page and mark related affected customers' ticket closed
Ontario Dildo Inspector
Miss you buddy.
I was monitoring everything all day waiting for them to return and go through to verify everything but then I spent the last hour accidentally coming across this.
Anyway, doing that now as planned. I specifically stayed up so I wouldn't miss it, going to go through all the nodes and validate everything's okay since things may be funky after a reboot. Might need another reboot for some of them just to ensure everything's good since some of these haven't rebooted in a while. I do see some of them are already running into issues so going over those first.
All good now, except missing Phoenix Debian template.
NYCB030 and NYCB043 are back online!
Fast as fuck Core i9 VPS (aff) | Powerful AMD Ryzen VPS (aff)
LEB strikes again with the "Expect VirMach to deadpool any minute now" saga. I find it odd that LEB is still writing about you. Were you once married to someone there ? Because the way they write about you is like an ex-wife/lover who is really pissed off. Tell the truth, did you and Jon have a thing going on the side ?
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Looks like all of my VPS from VirMach is online again, which were affected in the NY datacenter outage.
(My VPS's from DediPath (directly) & from Ethernetservers are also online - they use the same DC).
It will be very hot this weekend, backup is therefore needed
Virmach Deals
Wait, even vir chap'd got wives
Chinaman received a crit blow
Ontario Dildo Inspector
NYCB027 node remains locked.
It does not seems like simple reboot will solve NYCB028 issue as everything in NYC was shut down anyway.
Hi all, anybody on NY14 ? get these messages at boot regardless if you try to boot a CD or from HD.
ny 022 is not work Offline PhysicalValuable-VM id：688689
Click on also can not be normal
↑↑↑ another victim of that dump panel
Ontario Dildo Inspector
This is the #1 issue and #1 reason we get tickets recently. People have an old ISO mounted, probably forget about it, after it gets rebooted a long time later, especially if we updated the ISO since then, SolusVM will not unmount it automatically and instead completely refuses to boot the VM back online without providing proper error on clientside.
I made a knowledgebase article for it already but all you need to know is you have to unmount ISO and then boot.
All me, I did it after watching a few customers explain how they understood things. It's specifically tailored to reduce unnecessary support tickets but of course as with all changes it's had the opposite effect for the time being.
Biggest confusing thing for people I guess was hiding the password field, too many people relied on it I suppose which isn't good since you're supposed to grab it and then change it for security reasons. I'll see about adding that back in under the reset password section where it shows it. Any other complaints you have about it let me know.
Do you mean to reinstall the system? oh god my data
What? No.
I have no ISO mounted , after the power failure in NY it just don't want to boot anymore from whatever device HD/CDROM
If i mount a ISO it even crash and shutdown the instance too, VNC on or off all does not matter.
Can you assist me with the operation in the background?
Reboot in 15 minutes, I'm taking a look at NYCB014 now and fixing any problems.
OK，is work
All fixed thanks !