VirMach
VirMach
About
- Username
- VirMach
- Joined
- Visits
- 3,207
- Last Active
- Roles
- Member
- Karma
- 9,672
Comments
-
(Quote) First batch is in NYC. Second batch SJC. They're all for the most part racked, even the batches we haven't received yet, but they just need to be set up. I had this written out, maybe I forgot to post it? Around 15% are immediately…
-
(Quote) We do have some tickets from before we sent out the emergency notice, and some people who aren't as caught up to the series of events that occurred so it's best to specifically ask for new server details and mention your requiremen…
-
Just throwing this out there as well: Ryzen virtual servers are not going to be affected by this matter. SolusVM and WHMCS (VPS control, billing databases) are safe as well. We're wrapping up any leftovers today and looking at disaster recover…
-
(Quote) In our next meeting with the legal team, I'll attempt to get back some information for you. We will of course pursue this to the best of our ability. I would not necessarily know what legal ramifications are available, if any, that wou…
-
(Quote) Are you on a dedicated server that's currently offline?
-
I'm working with our developer today to pro-rate refund all dedicated servers where possible, for any remaining duration. We don't want to give people the wrong idea of what's going on and want to reduce any panic, at least on any end…
-
(Quote) I wouldn't be able to comment on this. (Quote) We have a contingency plan. It's not perfect. Nor were we expecting it to go this far or be done in this way. I've been working on it the entire time, and have emergency replacem…
-
(Quote) "There's been a lot of cases of people doing their own network configurations due to issues with reconfigure button, but then they set the IP address to the node's main IP or a random IP in the subnet. This is considered IP st…
-
Everything other than LAXA014 should be at the very least temporary resolved. I'm compiling a list of more permanent solutions to implement and still investigating some nodes to see what happened. (Quote) Are you on Ryzen?
-
(Quote) Message me the IP, I want to make sure it's not a node-wide issue.
-
(Quote) Yeah, possibly IP conflict. > @cybertech said: (Quote) It's actually a pretty hilarious paradox. They keep raising the rates, reducing the level of service they provide, and I highly doubt it translates over to the hands getting pai…
-
(Quote) Yeah, possibly IP conflict. Enabled protection on it. Reconfigure and try again. If it doesn't work, let me know. This could also potentially knock out the entire network for NYCB018 if we're extremely unlucky so I'll check ba…
-
(Quote) All labeling ever did for us is cause MORE problems so I stopped doing it. Example A: * Nothing is labeled. * Techs set it up how they want, and maybe communicate it correctly if we're lucky. * Tech may be able to follow instructions …
-
(Quote) I haven't been billing for these recently but they also take an outrageous amount of time to complete, so a fair bit of warning. This is one of the things I've tried and failed to fix over the last two years. Maybe we'll final…
-
Finally fixed NYCB036. They had the LAN port plugged in as dedicated IPMI on the switch we use for IPMI, and labeled it as a port on the main switch when it wasn't. Took a fair bit of detective work to find since the whole setup is a mess at t…
-
(Quote) This is one of the things where I don't even think it's their fault but they're on a blacklist UCEPROTECT which as @FrankZ mentioned should not be trusted but for some reason it's "industry standard" for a bunch…
-
I got lucky with SJCZ004. After about 20 reboots and messing with the BIOS, when I finally gave up and was going to try recovering data, I selected virtual disk on BIOS and it booted into the OS instead (???) seems like it just wants to do the oppos…
-
(Quote) I've gone down these dozens of times at this point and checked for broken IP assignments, looks like I missed one on NYCB018 because I'm semi-dyslexic and it was .212 instead of .221 on a single VM. I've corrected that one now.
-
Announcement: We've added an addon for all the people over the last few months that have indicated their extreme displeasure with their service going offline, claiming that as a result they lost thousands. There have also been people that were…
-
(Quote) Yeah, I saw that as well as a few other things happen and at this point I'm trying to figure out if I actually need to just start a VirBot cloning program so we can ship me off to live in a cage at every datacenter.
-
Hivelocity's the one that acquired Incero right? I don't know why they didn't come to mind (HV) when I was thinking about Dallas. We can't do it right now but there's absolutely no way we can stay with Flexential at this sta…
-
Holy crap, we're actually so doomed in Dallas @FrankZ they finally got back to us and said (Flexential) due to the "complexity" of the request, the request being: * Clear CMOS * Reset BMC I think maybe I made it sound so complicated…
-
(Quote) Yeah that's already been established but at this point I've basically made myself step back and cool off, otherwise I feel like we're going to be migrating everyone around for the next 2 years before we can settle down with an…
-
Chicago going up today.
-
(Quote) Re-enabled.
-
Finally got back the physical port numbers for the servers I needed, NYCB036 (formerly 101) should be fixed soon, just need to wrap up what I was working on and configure it in the switch. SJCZ004 unfortunately still down, I was going to fly in yest…
-
(Quote) Took it away while we do more migrations, we can't have it drastically change too much or it'll put the plans out of whack. It'll re-enable later tonight.
-
(Quote) There's no way to differentiate between fully broken and partially broken nodes to display it only to those people.
-
(Quote) Button updated. I can't get it to work but let's see if anyone else has any luck (edit -- looks like it did work for at least one person so far.) To be far I was trying to break it by being impatient and refreshing/closing it. Let…
-
(Quote) I'd actually accept these kinds of request if it was possible to keep it clean but realistically that means you get a new VM and then the old one just kind of hangs around while it's offline and takes up space until we manually ver…
-
(Quote) How do you propose a migration of an offline server?
-
Yeah I don't know how reliable this random site is that I found but: (Image)
-
(Quote) Phoenix is completely trashed right now.
-
(Quote) LA10GKVM14 was part of a massive PDU or power circuit event with them, where like a dozen plus servers went offline and online, and then offline, causing a bunch of power supplies to fail. A lot of times when we've had power issues like…
-
(Quote) Honestly at this point I think 8.8.8.8 issue is actually not on any specific datacenter's end. We had it happen on our WHMCS for a little bit, and that one's not even hosted with any datacenter we use right now for these Ryzens. We…
-
NYC storage node, I had to do a lot of weird setups for this one to make it work as we faced problems such as a disk going missing, the wrong switch, and so on in the background. Right now it's LACP aggregated 3Gbps. Originally was supposed to…
-
(Quote) Very unlucky user or very lucky user based on preference, when it happened, etc. Maybe your services are getting migrated to Ryzen, maybe they're all broken.
-
I was going to get a flight to San Jose for tomorrow morning but I'm still waiting to see how we can even get DC access since it's the first time. I might still go on Tuesday and see if I can beat the DC hands to it for SJCZ004 at this poi…
-
I did also finally get some switch access, so you will either see more nodes up in NYC that we can use for migrations, or you'll see me break the networking.
-
(Quote) All systems are existing systems that have been tested, some more than others. The newer nodes I've actually spent less time testing, not as a result of being careless, but in that a lot of the issues were already ironed out so it was a…
-
Okay after working on this all day, outside of the two nodes somehow still down (SJCZ004 and NYCB036) I've counted these specifically and we've gone down from something like 2% of people after the most recent round of migrations to exactly…
-
(Quote) There's about 3-5% of the VMs offline, but the node as a whole isn't. Those are on our queue.
-
(Quote) It's been renamed to NYCB036 and I've been painfully aware of it being down. I have no access to IPMI and no one seems to care. I've mentioned it at least 4 times and it hasn't even been marked as request put in to the DC…
-
(Quote) 04:40:12 up 6 days, 4:12, 1 user, load average: 8.04, 8.25, 7.43
-
(Quote) We completed PSU swap here, so we have to see what's wrong at this point. It's just shutting off and not throwing out kernel errors so what's left that could be possible I can think of is CPU suddenly overheating or loose cabl…
-
SEAZ008 finally back up, god that took a long time. DC hands plugged in IPMI to LAN and LAN to IPMI, finally corrected. A lot of others have been fixed as well, all of Atlanta (DC hands gave the port numbers in reverse order) and all of NYC by now e…
-
(Quote) I'd scream at CC but I've given up on that. They placed a permanent nullroute on the main IP. This is probably the 30th time they've done this, from a single website being malicious. I'm trying to get a server up and just…
-
(Quote) Not sure what you mean by that, on my end the /24 is functional and previous issue(s) corrected. I actually fully forgot what happened with this one, my brain can only store so much when it comes to literally everything breaking at some poin…
-
(Quote) Expected. This is one of the reasons I said IPv6 may break and it's not yet officially delivered and we put it on the backburner for some locations. It was only set up to function on the same VLAN, we have to go through and split it up …
-
It's crazy how hard the NIC was trying to keep up with the previous network configuration, this screenshot says it all. I almost had to check and make sure the change didn't just knock everyone offline or crash something. (Image)