@ZA_capetown said:
Yay for the progress. And the weird keeps life 'interesting' discoveries :-D
Congrats. :-)
@Virmach how far away do you live / how much travel time to the Oklahoma datacenter is it for you?
Is it a case of whenever something goes wrong it would be easier to just jump in your car and be there in 15 minutes, rather than ask datacenter hands to check things out?
Or not exactly that conveniently close by?
I live closer to the datacenter than the two main people that work there. Once my car window rolls back up during thunderstorms I’ll be unstoppable.
@VirMach said:
I have two stories for you guys, backstories for two servers we got back.
First let's talk about the miracle of MIAZ014. I had my noise cancelling headphones while packing the servers in Miami, so I did not notice this at all: the tech in Miami, after some maintenance, had unscrewed the motherboard and left it off. Of course, when I get the box, it also happens to be the one that's royally crushed, it looked like the UPS guy stomped on it (maybe to make the rattling stop.) Other ones, fine.
I get it back and I have my noise cancelling headphones on (wow, they're really good at cancelling noise.) Since the chassis was crushed, it had pinned the motherboard. Immediately I thought it was toast, and I just kind of hammered the chassis back in shape, figured we'd use it for something else. For some reason, this ended up in the wrong server pile and it made its way to the datacenter. It got racked, but then I noticed it'd beat up and remembered the name. Just for fun, I decide to try to power it on. It doesn't power on, no surprise. I'm about to just unrack it and take it home, and that's when I first notice it rattling. I open it up, and see the mess. NVMe SSDs everywhere, crushed heatsink, yeah no wonder it didn't turn on. Except I notice the tech also didn't plug back in the power pins, he just plugged it back in and it was set to auto boot so he called it a day and it was just running since, for like a year or two. So I plug in the power pins, and it boots. Everything's fine. No data loss, nothing. I'll still do testing and replace some things after we get any customers off.
Then let's talk about the most infuriating story every, Dedipath and the curse of Ernie. For this one I'm going to try to word everything relatively neutrally and let you make your own assumptions on what really happened.
At the beginning, for some reason, we had a lot of trouble with the servers sent to Dedipath. Pretty much in every location in some way, it was weird. Like lots and lots of troubleshooting just to get them going even though they were tested before being sent in. There were many servers, but there were a pair of twins in Atlanta that were really giving us a headache, and they seemed to regress as they were worked on. So we'd ask them to troubleshoot, and then it wouldn't improve and now it wouldn't even post, etc. I do not remember many of those details clearly, but there's one part of it I remember very clearly, and it's the reason I didn't even bother touching this server since 2022. It's just been sitting there until today.
I still have some of the chat logs/communication on these two servers so I was able to verify and make sure I'm not misremembering. After all this work, we sent in a board replacement for one of these. They allegedly swapped out the board, but it had the same exact problem. Strange, must be what's on the board then even though that didn't make much sense after the details we collected while troubleshooting. So we sent in memory replacement. more troubleshooting, we also sent in another set of memory for the other one. I got back the potentially bad memory, and it was fine. So I built out a new server, we didn't have a chassis readily available, but think of it like a motherboard with everything attached, tested, wrapped, and sent in ready to do a transplant. The maintenance was done, but we had the exact same issue still. I think it was a total of two sets of memory and motherboards for one, then three other motherboards and 5 sets of memory for the other, plus NVMe SSDs, etc.
This is where Ernie comes in. I'm still trying to figure out what's going on with everything, because it doesn't seem like a swap was done, I'm getting different information from different people. I ask Ernie to make sure maintenance is being done properly on it, ask him again if the board LED is on, if it's responding to being powered on, what the board is displaying. No answer on any of that, but he says something really interesting. For these servers, Ernie said he personally flew to Atlanta, swapped the RAM, and reseated the CPUs. He specifically also mentions he also reseated the RAM and made sure it's in and they're "simply giving no output." That they've exhausted their capability, they need to be sent back for repair, and it's going to be expensive to keep doing remote hands on them. "I have very little faith they will work." Finally, he ends it by saying:
"I'll be very candid I'm not sure what vendor is doing your initial builds but Frankenstein on some of these is an understatement. I do have a source for these to properly be built if you want a intro."
I let him know I'm the vendor doing the builds, reiterate everything above. He says the board swaps were done, etc, then follows it up by saying the most recent motherboard that was sent in to fix the issue was placed in storage. Then he says it might have been put into recycling. That doesn't make sense but I'm ready to just get whatever back. He offers to credit us $540 for all this, which I don't know if we ever got but we definitely got a $550 or so bill just to have these sent back plus a few thousand in RH, plus we lost all that hardware.
One of the servers we get back is one of Dedipath's servers. I already know what's about to happen but I let Ernie know, and of course we send back their hardware (and never hear back on our server they sent wherever instead.) So we lose another server.
The one I did get back, that's been sitting in a corner, I finally open it today. The one Ernie confirmed he worked on personally.
First of all, there's an NVMe missing. Whoops I wonder what happened to that, weird. Two DIMMs are not seated properly and when I say not seated properly what I mean isn't that it didn't "click" but that whoever worked on it pressed in one side, and didn't even align the other side at all, so it's kind of hanging sideways, visibly and the other the same but it got pushed in a bit more.
The heatsink seemed to be a little loose but honestly I can't really confirm that part 100% it could just be I'm tired and thought it opened too easily. The CPU isn't even in, it's stuck to the heatsink but to be fair this is a potentially known problem and a reason the CPU would have been opened up (to fix.) Except there's thermal paste EVERYWHERE like on parts of the heatsink that don't even go above the CPU, on the board, UNDER the CPU (this isn't the first time this has happened, to be fair. No wait actually this was in 2022 so it likely was the first time this happened.)
I cleaned up the paste, pushed in the memory, hit the power button and it works fine. Took maybe 3 minutes. And that's how we lost... I don't know, what's the tally? The end.
This all makes me so much more excited for Oklahoma. Speaking of which, I'm finally about to head to the facility. Raining stopped several hours ago, I'm just trying to take down a big batch of servers without forgetting anything.
its really bad luck, but any way to prevent this in future?
I mean it kind of took care of itself, they can’t hurt us if they’re out of business. I do want to give a huge shoutout to Ivan at LAX though he was an amazing tech and I hope he found a great position somewhere else.
Also to kind of balance out these two stories, this isn’t much of a story, but xTom moved our equipment to the new facility and it felt as if it never happened, perfect move, way better than I did at LAX.
I’ve relocated the memory for the storage server for probably the 5th time. I need to stop walking around with it in my hand. Now time to relocate the 16GB DDR4 that I need for that other specific server.
Someone remind me to put a camera in every room and feed it to ChatGPT to keep track of items. Or maybe I’ll just condition myself to walk around holding dumbbells then one day I’ll also be able to pick up more than two or three servers at a time again.
Hopefully I’ll actually get to leave soon. But hey good thing this time I have a proper checklist.
The worst part is the second I remember I have to do something else I just leave whatever I’m holding in the nearest shelf and there’s plenty, so I have to scan up and down 6 feet for it.
(Edit) casually just accidentally found the 8TB U.2 NVMe that I’ve been desperately trying to find for over a week.
@yoursunny I’ve located a box that has your username on it, let me know if you still want it and I should be able to send it out within a couple more years as long as you still live there.
@VirMach said: @yoursunny I’ve located a box that has your username on it, let me know if you still want it and I should be able to send it out within a couple more years as long as you still live there.
@VirMach said:
Alright heading out now, I’ll stay there until I get the network up.
please dont leave until the migrations are complete 🤭
Okay deal, unless my hotspot gets any worse than it already is right now.
Awesome news by the way, I got the connectivity up between the router and switch. I must have just been sleep deprived last time. Well I'm also sleep deprived now but perhaps it helps that I worked on this first thing when I came in rather than racking a bunch of servers. Other theories include: 1) I was starving last time but I had a sandwich this time, 2) I didn't get any smarter, I just got lucky this time, 3) it's Thursday and I realized we're about to fall behind another whole week, 4) I didn't want to get stuck in the datacenter forever per my last promise.
Bad news though, the server it's meant for needs a specific Epyc processor that's high core count and while I have at least two of those and it's a huge CPU, they look very tiny next to 200-300 servers worth of equipment and junk. I have to locate the processors now because it's not in the motherboard like I thought. There's also a second 8TB that I have to find unless we've accepted it's not going to be RAID1 but the original plan was RAID1 and I did also locate something like 5 controllers specifically designed for U.2 as well right next to the 8TB drive. Hey at least I'm pretty organized, everything's in little related clusters but in random spots. It's like a VirMach universe and we just have to locate all the galaxies.
With all that said, this is not meant for an @imok 1200GB special.
@yoursunny said:
JoshIdeas is deadpooling.
We have 1C 2G 40G for $6/year in IAH location.
Will you offer refuge dealz in OKC?
IPv6 only, we could do $6 per year or $9 per 2 years for 2C 4G 40G but only if 20,000 people are interested at once. Let me know after speaking with them all and confirming.
@yoursunny said:
JoshIdeas is deadpooling.
We have 1C 2G 40G for $6/year in IAH location.
Will you offer refuge dealz in OKC?
IPv6 only, we could do $6 per year or $9 per 2 years for 2C 4G 40G but only if 20,000 people are interested at once. Let me know after speaking with them all and confirming.
I think I should probably leave as I've been here for 18 hours and honestly that many hours of work didn’t get done this time around, I had to end up dealing with some other things just with terrible internet access.
I'm invoking this loophole to leave.
@VirMach said: Okay deal, unless my hotspot gets any worse than it already is right now.
I’m out of the building now and it’s still <1Mbps.
The router had to be prepared for being closer to production and that caused some announcement issue, that has to be resolved and once that’s done we have some servers going automatically online as the rest is complete. As in, it was complete at some point in time today, maybe 8 hours ago. Software update, finalizing failover, rebooting and some security.
At this point it’s not a major problem and easy to resolve but I just want to drink water, use the bathroom and maybe sleep.
@VirMach said:
I think I should probably leave as I've been here for 18 hours and honestly that many hours of work didn’t get done this time around, I had to end up dealing with some other things just with terrible internet access.
I'm invoking this loophole to leave.
@VirMach said: Okay deal, unless my hotspot gets any worse than it already is right now.
I’m out of the building now and it’s still <1Mbps.
The router had to be prepared for being closer to production and that caused some announcement issue, that has to be resolved and once that’s done we have some servers going automatically online as the rest is complete. As in, it was complete at some point in time today, maybe 8 hours ago. Software update, finalizing failover, rebooting and some security.
At this point it’s not a major problem and easy to resolve but I just want to drink water, use the bathroom and maybe sleep.
dont forget to pace yourself; it's marathon you're running.
@VirMach said: .. as I've been here for 18 hours and honestly that many hours of work didn’t get done this time ..
10 hours too long, without an extended break - it becomes counter-productive. Sleep deprivation can only happen for so long, before your mind & body suffers.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@taizi said:
what datacenter is it? i barely heard who selling Oklahoma server,because the network will route to dallas first,and the price also higher than dallas
Early comparison. I'm comparing it with Dallas CDN77/DataPacket. Average column, rounded. Major cities only, up to first 2 per city. Skipping Dallas (because obvious.) I don't know why I'm complicating this instead of just showing the ping.pe I guess I want to let you guys do that yourselves and to keep it as notes for myself. Where OKC does better marked as ⬇️
OKC, Dallas | FROM
61, 74 | Vancouver Telus ⬇️
69, 84 | Vancouver Shaw ⬇️
44, 43 | San Francisco DIgitalOcean
39, 31 | Los Angeles Hurricane
40, 31 | Los Angeles DMIT
48, 58 | Seattle Google ⬇️
24, 18 | Denver Cogent
23, 17 | Denver NetActuate
26, 11 | Kansas City Rozint
29, 25 | Chicago FDC
26, 25 | Chicago NetActuate
36, 33 | Buffalo ColoCrossing
40, 36 | NYC Hurricane
42, 45 | Montreal OVH ⬇️
144, 141 | Sea Paulo NetActuate
145, 200 | Sao Paulo Hyperfilter ⬇️
107, 102 | London FDC
111, 112| Paris Online.net ⬇️
112, 106 | Paris NetActuate
Okay too many major cities coming up. Quick note though, OKC does worse to Netherlands, better to Frankfurt, better to Romania. Let's do Asia, and skip over middle east as I think they both just do terrible here.
207, 203 | Singapore
174, 138 | Tokyo DODO
134, 137 | Tokyo Vultr ⬇️
184, 222 | Sydney Vultr ⬇️
189, 178 | Taiwan Google
China is all over the place, so I'll just do it this way:
OKC is currently much worse to Shandong China Unicom, 20% loss and 301 average
OKC is second worst to Lishui China Unicom, 13% loss and 274 average
Dallas has Jiangsu China Telecom 100% loss, Chengdu 292 average worst
OKC is better to 11, worse to 3
OKC best is 171 to Tencent Shanghai, 3% loss
Dallas best is 207 to Lishui China Unicom, 0% loss
Some notes:
Cogent - I've noticed most the routing above that's worse than Dallas is because it ended up going through Cogent. Once we do some magic to try to get that to be used as little as possible on those routes it should improve and to the level its possible, I feel like OKC has a pretty good chance of being better than Dallas to most places.
A perfect example of Cogent causing problems is Kansas City. Notice how if you open a map, Kansas City is much closer to Oklahoma City but it's 26ms from OKC and 11ms from Dallas. Cogent routes it in a weird circle because they're missing an important connection.
For anyone confused on the Cogent note above since we don't actually have Cogent yet, remember we're comparing incoming, not outgoing.
San Jose seems to be the point where Dallas and OKC currently match, they're only off by 0.2ms so anything north of that, good chance it's better from OKC, anything south, better from Dallas. At least for west coast.
@VirMach said: .. as I've been here for 18 hours and honestly that many hours of work didn’t get done this time ..
10 hours too long, without an extended break - it becomes counter-productive. Sleep deprivation can only happen for so long, before your mind & body suffers.
The part I left off is that I didn't sleep between my last visit to the facility either so I was actually awake for a little over 3 days. It's just hard to sleep sometimes when a lot of exciting work is getting done. Don't worry though, I slept for 22 hours after returning home.
Fixed previously mentioned router issue after it was finalized. I have a checklist of things to grab before heading in one last time (hopefully) before we have at least one server ready for migrations. That means barring some new problem, we are extremely close to one being up. I think what's needed is just verifying the few servers I mentioned as ready have the correct IP configuration or rebooting them.
I'll probably be here for a couple more hours then will head there. I'll see how plausible it is to try to also get in some Ryzen bare metal.
Storage node is ready, I need to take down a couple SSDs to install OS and finish RAID configuration. All drives healthy.
Epyc Windows node is ready (with the reduced amount of memory as previously mentioned.) Needs OS.
Several Ryzen VPS nodes are ready (over a dozen.) With OS. Need to be added as slave nodes.
@VirMach said: I forgot to drink my coffee before I left, I made it and everything but its sitting on the kitchen counter. I'm devastated.
Just order one with delivery, no reason to suffer.
I remember, you were kinda coffee-geek iirc and you didn't like the ordering one as it was dog shit.
RIP
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
@VirMach said: I forgot to drink my coffee before I left, I made it and everything but its sitting on the kitchen counter. I'm devastated.
Just order one with delivery, no reason to suffer.
I remember, you were kinda coffee-geek iirc and you didn't like the ordering one as it was dog shit.
@VirMach said: I forgot to drink my coffee before I left, I made it and everything but its sitting on the kitchen counter. I'm devastated.
Just order one with delivery, no reason to suffer.
I remember, you were kinda coffee-geek iirc and you didn't like the ordering one as it was dog shit.
RIP
I can't leave the datacenter, security is too high someone has to come back and scan their body parts to let me in.
I just want everyone to know this is the temperature in the hot aisle right now right behind our cabinet. We need to get some people running benchmarks ASAP or I’ll freeze to death.
Yes I really brought a smart thermometer with me I wanted to make sure my jacket was justified.
@VirMach said: Fixed previously mentioned router issue after it was finalized. I have a checklist of things to grab before heading in one last time (hopefully) before we have at least one server ready for migrations. That means barring some new problem, we are extremely close to one being up. I think what's needed is just verifying the few servers I mentioned as ready have the correct IP configuration or rebooting them.
Reason these didn't come online straight away is because they're using the Miami IPv4 subnets which had an issue. I only tested the original subnets we had for Oklahoma when verifying the other day. While we wait on that, I'll be twiddling my thumbs. Okay no I'll find something to do, more cage nuts and maybe get everything connected in the second cabinet and get @Python's VPS back on.
@VirMach said: this is the temperature in the hot aisle right now
Totally tropical, man: 3degC above temperature here today!
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Dang that’s cold for long hours of work @VirMach must really miss that coffee now
@VirMach said:
I just want everyone to know this is the temperature in the hot aisle right now right behind our cabinet. We need to get some people running benchmarks ASAP or I’ll freeze to death.
Yes I really brought a smart thermometer with me I wanted to make sure my jacket was justified.
The first person to open a migration request has been moved to Miami (this is an inside joke between me and the one person that's been migrated, you guys wouldn't get it. )
Don't forget (I'm speaking with the one person who was moved) there's going to be some unannounced maintenance. Everyone else, no need to worry, you are likely not the one person who had their service moved and therefore there's no planned unplanned maintenance for your service and it is safely still in the same location.
Once again I'm only speaking with the person who had their service migrated, but also keep in mind that some things may change like your service's IP address when everyone else joins the party. I'm going to go painfully slow and go in the order they were put in.
@animeiscool said:
I just got my VM migrated to OKC
I don't want this person to feel uncomfortable for being the only person who's been moved so I won't name them.
I still can't locate our 7542/7452. I don't know if the boards we have are compatible with older generation, but I have a bunch of 7501's that I may just put in so we have them as a placeholder for now and then it's just an easy swap later down the line.
I had a lot of updates written but some were outdated and then it got long, I'll just post a few snippets here and there.
Ryzen Bare Metal
It's looking like this right now. Can change, it's already changed so much, I just never updated you guys until I was semi-sure.
The prices are estimates and are the sale prices. What will likely happen for the sale is that I'll build everything out, won't put in specific drives, and they'll be "pre-orders" except not really a pre-order, as in it'll only be done when they're racked, it would just help get an idea on the demand and then let's say 72 hours setup time or we say we failed to deliver it and just refund you.
Then, expect there to be non-early birds on them but maybe $5 a month higher after X quantities sold. MSRP starts at $49.
A ) $25/mo - Ryzen 5500*, 16GB RAM, 4x1TB HDD (Can optionally upgrade to 4x3TB HDD) w/ a 500GB SSD snuck in as optional boot drive?
B ) $35/mo - Ryzen 5500, 32GB RAM, 4x500GB** SSD (Can optionally upgrade to 4x1TB SSD)
C ) $45/mo - Ryzen 5500, 32GB RAM, 2x2TB NVMe (Can optionally add up to*** 4x hotswap disks)
D ) $55/mo - Ryzen 5500, 64GB RAM, 2x4TB NVMe (Can optionally add up to*** 4x hotswap disks)
Notes:
Non-ECC memory, no IPMI. KVM will likely be optional and $15 or so setup fee and/or $1-2/mo. We can technically do ECC and IPMI, for +$40/mo but I doubt anyone would want that.
They'll likely come with a /30 IPv4 and /48 to /64 IPv6 (probably /56.) Higher amount available for IPv4 but not at a price that'd be attractive for the sale.
*These are the ones that may switch over to Ryzen 2600's in the future. Not retroactively, so if you buy a 5500 and keep it, then it'll be left alone. These boards technically do the best on 3000 series so we'll see how it goes and a third version for processors might be introduced later down the line, but it's going to be 5500 for the first 20 to 30 of them.
**Might end up being 4x1TB SSD, depends on how many of each we have available. I remember them as being mostly 500GB but all the drives I've pulled out of storage have been 1TB SSDs.
**Haven't tested the actual limits, likely capped if we want it not running into issues. I actually don't even remember how many SATA ports we have on the boards, I could check in 15 seconds and not write all this out but I won't.
Let me know if there's any feedback on how these should be modified, if pricing looks good, and so on. We're pretty flexible with them at this stage. All I know is that I tried to put in both NVMe and a RAID controller and while that's technically possible, it would need to be done later with some custom bracket, specific controllers, etc.
@VirMach said: Non-ECC memory, no IPMI. KVM will likely be optional and $15 or so setup fee and/or $1-2/mo. We can technically do ECC and IPMI, for +$40/mo but I doubt anyone would want that.
really want ECC for better production environment(mY biLLioN BuSIness),and also want IPMI/or other controller like PiKVM
+$350 ONE-TIME
I remember we have mentioned that,maybe give us some discount by pay the ONE-TIME fee?
@taizi said: really want ECC for better production environment(mY biLLioN BuSIness),and also want IPMI/or other controller like PiKVM
PiKVM would cost more than one-time setup for a dongle, but the dongle is going to a pretty old multi-port KVM unit with limited simultaneous seats and also old enough to where there's no HTML5. We can pretty much do anything... but it will add to the cost. So PiKVM or newer multi-port switch plus dongle would be an extra $20 ($15 versus $35 setup.)
@taizi said: really want ECC for better production environment(mY biLLioN BuSIness),and also want IPMI/or other controller like PiKVM
ECC can technically be done if you downgrade from 5500 to 2600 or 3600 (if we get any) for an extra $10 a month or so. It's not something we'd likely be offering but if you end up being interested I can customize it, and you'll need to get the 64GB memory server because the ECC sticks are larger. Or with a board that supports IPMI, around +$40 a month as mentioned (or $400 one-time.)
But with all that said, it sounds like you just need an Epyc server. By then it'll be cheaper than whatever mess this would be... BUT, we could do it if you really want a Ryzen. I can even do a hacky way of getting ECC to work with 5000 series. But it's hacky.
Epyc server would end up being around $59 a month... BUT... it's limited on disk. Since it'll be uses U.2/Enterprise. So something like 1TB base, $10/mo or $50 setup per additional 1TB or so, super rough pricing on that.
@taizi said: I remember we have mentioned that,maybe give us some discount by pay the ONE-TIME fee?
I feel like the prices already include those discounts, without the $350 one-time fee. I guess it's not a 5600X, it's a 5500, and it's already past the amount of NVMe included for that $350 one-time upgrade.
@taizi said:
oh wait,is that general offer?was though it is for old E3 customer
Oh maybe I should have mentioned this last part:
Anyone who previously had a dedicated server with us and was impacted by a delay in it being switched to Ryzen and you wanted Ryzen, or you were negatively affected by the time we got Biloh'd or the time we got Ernie'd then we would offer you an equivalent Ryzen bare metal at the same price you were paying. Unfortunately, since I know someone may ask, this doesn't mean we can include ECC memory or KVM or IPMI or more IPv4 even though the original technically had those thing(s.) Those are all additional line items and weren't calculated in my ability to make that offer. Also putting another disclaimer, this doesn't apply to any dedicated servers we may have given out for free. Maybe. Let's say all of this requires my approval but I'll be nice.
It'd be $200-400 one-time setup fee or something like $15 a month. The IP addresses won't be cheap on dedicated servers. The lowest I could see a /29 being would be $8 per month additional. Depends on how many I set aside for this, but at the beginning we're assuming no one wants it. So if more people want it, price would come down.
People can send in their own drives and I'll install it without any additional setup fees (for members of this community, at least for these initial purchases) as long as you do not ever want it back, even if you cancel (I'll make an exception to this if you're sending us a $500 hard drive or something like that and get pre-approved for me to track it so we know it's yours.)
Extra drives, those can end up being pretty cheap. I have a lot of them, depends on what's left over. But probably... super quick estimate.
+$3/mo per 2-3TB HDD or 500GB SSD
+$5/mo per 1TB SSD
+$10/mo per 2TB SSD or 6-8TB HDD
@sh97 said:
What about a $100-169/y dedicated server, as an extremely early bird offer, limited to 1 or 2 qty?
If you had a server with us before, see above, it might end up being around $170/yr for the cheapest ones we sold in the past. Sadly your proposed pricing wouldn't even cover colocation costs and I'm not trying to lose even more money yet on this location, allow it to become profitable first and that sounds more like something we'd do if we have extra empty ones in the future.
I could definitely do $169/yr for first year only but I don't think you guys like those kind of deals. I'll see what's possible.
Comments
I live closer to the datacenter than the two main people that work there. Once my car window rolls back up during thunderstorms I’ll be unstoppable.
I mean it kind of took care of itself, they can’t hurt us if they’re out of business. I do want to give a huge shoutout to Ivan at LAX though he was an amazing tech and I hope he found a great position somewhere else.
Also to kind of balance out these two stories, this isn’t much of a story, but xTom moved our equipment to the new facility and it felt as if it never happened, perfect move, way better than I did at LAX.
I’ve relocated the memory for the storage server for probably the 5th time. I need to stop walking around with it in my hand. Now time to relocate the 16GB DDR4 that I need for that other specific server.
Someone remind me to put a camera in every room and feed it to ChatGPT to keep track of items. Or maybe I’ll just condition myself to walk around holding dumbbells then one day I’ll also be able to pick up more than two or three servers at a time again.
Hopefully I’ll actually get to leave soon. But hey good thing this time I have a proper checklist.
The worst part is the second I remember I have to do something else I just leave whatever I’m holding in the nearest shelf and there’s plenty, so I have to scan up and down 6 feet for it.
(Edit) casually just accidentally found the 8TB U.2 NVMe that I’ve been desperately trying to find for over a week.
@yoursunny I’ve located a box that has your username on it, let me know if you still want it and I should be able to send it out within a couple more years as long as you still live there.
Alright heading out now, I’ll stay there until I get the network up.
Yes, it's the XPG drive you promised us three years ago.
https://lowendspirit.com/discussion/comment/89977/#Comment_89977
We have been waiting for it to upgrade our desktop.
We'll ring the 308 buzzer every time we go to Los Angeles until we receive the box.
We accept Karma donations for the last flan. 🍮 affbrr
@imok TIME TO ASK THE QUESTION!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
please dont leave until the migrations are complete 🤭
I bench YABS 24/7/365 unless it's a leap year.
Okay deal, unless my hotspot gets any worse than it already is right now.
Awesome news by the way, I got the connectivity up between the router and switch. I must have just been sleep deprived last time. Well I'm also sleep deprived now but perhaps it helps that I worked on this first thing when I came in rather than racking a bunch of servers. Other theories include: 1) I was starving last time but I had a sandwich this time, 2) I didn't get any smarter, I just got lucky this time, 3) it's Thursday and I realized we're about to fall behind another whole week, 4) I didn't want to get stuck in the datacenter forever per my last promise.
Let's see what other issues await us.
Bad news though, the server it's meant for needs a specific Epyc processor that's high core count and while I have at least two of those and it's a huge CPU, they look very tiny next to 200-300 servers worth of equipment and junk. I have to locate the processors now because it's not in the motherboard like I thought. There's also a second 8TB that I have to find unless we've accepted it's not going to be RAID1 but the original plan was RAID1 and I did also locate something like 5 controllers specifically designed for U.2 as well right next to the 8TB drive. Hey at least I'm pretty organized, everything's in little related clusters but in random spots. It's like a VirMach universe and we just have to locate all the galaxies.
With all that said, this is not meant for an @imok 1200GB special.
JoshIdeas is deadpooling.
We have 1C 2G 40G for $6/year in IAH location.
Will you offer refuge dealz in OKC?
We accept Karma donations for the last flan. 🍮 affbrr
IPv6 only, we could do $6 per year or $9 per 2 years for 2C 4G 40G but only if 20,000 people are interested at once. Let me know after speaking with them all and confirming.
I'm interested!
waifuwall.com | Anime Girl Wallpapers - Uptime Status
Yes x 20000
I think I should probably leave as I've been here for 18 hours and honestly that many hours of work didn’t get done this time around, I had to end up dealing with some other things just with terrible internet access.
I'm invoking this loophole to leave.
I’m out of the building now and it’s still <1Mbps.
The router had to be prepared for being closer to production and that caused some announcement issue, that has to be resolved and once that’s done we have some servers going automatically online as the rest is complete. As in, it was complete at some point in time today, maybe 8 hours ago. Software update, finalizing failover, rebooting and some security.
At this point it’s not a major problem and easy to resolve but I just want to drink water, use the bathroom and maybe sleep.
dont forget to pace yourself; it's marathon you're running.
I bench YABS 24/7/365 unless it's a leap year.
10 hours too long, without an extended break - it becomes counter-productive. Sleep deprivation can only happen for so long, before your mind & body suffers.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@VirMach Has the routing of the Tokyo server been switched to NTT again?
T is posible
What locations have IPv6 these days, just AMS?
OKC
We accept Karma donations for the last flan. 🍮 affbrr
Don’t worry, I can still wait a couple more hours.
Stop creating all those earthquakes and tsunamis… patience!!
( ps hope all 🆗)
blog archives
Something I forgot to post from yesterday.
Early comparison. I'm comparing it with Dallas CDN77/DataPacket. Average column, rounded. Major cities only, up to first 2 per city. Skipping Dallas (because obvious.) I don't know why I'm complicating this instead of just showing the ping.pe I guess I want to let you guys do that yourselves and to keep it as notes for myself. Where OKC does better marked as ⬇️
OKC, Dallas | FROM
Okay too many major cities coming up. Quick note though, OKC does worse to Netherlands, better to Frankfurt, better to Romania. Let's do Asia, and skip over middle east as I think they both just do terrible here.
China is all over the place, so I'll just do it this way:
Some notes:
The part I left off is that I didn't sleep between my last visit to the facility either so I was actually awake for a little over 3 days. It's just hard to sleep sometimes when a lot of exciting work is getting done. Don't worry though, I slept for 22 hours after returning home.
I'm not paying attention to that nor can I do much about it other than escalate it. If it stays like that, let me know.
Looks like an xTom change - GreenCloud is also affected.
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
OKC Update
Fixed previously mentioned router issue after it was finalized. I have a checklist of things to grab before heading in one last time (hopefully) before we have at least one server ready for migrations. That means barring some new problem, we are extremely close to one being up. I think what's needed is just verifying the few servers I mentioned as ready have the correct IP configuration or rebooting them.
I'll probably be here for a couple more hours then will head there. I'll see how plausible it is to try to also get in some Ryzen bare metal.
@VirMach It seems that the bandwidth for one of my VPSs hasn't reset as expected. Would you mind taking a look?
Taking a look now. Unfortunately I wasn't able to locate your VPS based on the provided information.
OKC Update
I forgot to drink my coffee before I left, I made it and everything but its sitting on the kitchen counter. I'm devastated.
I’ve already opened a ticket (#245771). Thanks in advance!
Just order one with delivery, no reason to suffer.
I remember, you were kinda coffee-geek iirc and you didn't like the ordering one as it was dog shit.
RIP
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
One can always have the #cold coffeve
I can't leave the datacenter, security is too high someone has to come back and scan their body parts to let me in.
I just want everyone to know this is the temperature in the hot aisle right now right behind our cabinet. We need to get some people running benchmarks ASAP or I’ll freeze to death.
Yes I really brought a smart thermometer with me I wanted to make sure my jacket was justified.
Reason these didn't come online straight away is because they're using the Miami IPv4 subnets which had an issue. I only tested the original subnets we had for Oklahoma when verifying the other day. While we wait on that, I'll be twiddling my thumbs. Okay no I'll find something to do, more cage nuts and maybe get everything connected in the second cabinet and get @Python's VPS back on.
Totally tropical, man: 3degC above temperature here today!
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
can you run this script?want to check the actual net quality
bash <(curl -Ls Net.Check.Place) -Efrom https://github.com/xykt/NetQuality/blob/main/README_EN.md
I just want to know when tyoc27 will come back, I haven't contacted him for several months
I tried to launch it last week and found that it worked, you can try it too
@cybertech has been summoned…
Dang that’s cold for long hours of work @VirMach must really miss that coffee now
blog archives
It's time to renew this that's been idling since 2021. $6.30/y
Let me see if any of mine have been moved, I'll help out!
I just got my VM migrated to OKC
The first person to open a migration request has been moved to Miami (this is an inside joke between me and the one person that's been migrated, you guys wouldn't get it.
)
Don't forget (I'm speaking with the one person who was moved) there's going to be some unannounced maintenance. Everyone else, no need to worry, you are likely not the one person who had their service moved and therefore there's no planned unplanned maintenance for your service and it is safely still in the same location.
Once again I'm only speaking with the person who had their service migrated, but also keep in mind that some things may change like your service's IP address when everyone else joins the party. I'm going to go painfully slow and go in the order they were put in.
I don't want this person to feel uncomfortable for being the only person who's been moved so I won't name them.
I still can't locate our 7542/7452. I don't know if the boards we have are compatible with older generation, but I have a bunch of 7501's that I may just put in so we have them as a placeholder for now and then it's just an easy swap later down the line.
I had a lot of updates written but some were outdated and then it got long, I'll just post a few snippets here and there.
Ryzen Bare Metal
It's looking like this right now. Can change, it's already changed so much, I just never updated you guys until I was semi-sure.
The prices are estimates and are the sale prices. What will likely happen for the sale is that I'll build everything out, won't put in specific drives, and they'll be "pre-orders" except not really a pre-order, as in it'll only be done when they're racked, it would just help get an idea on the demand and then let's say 72 hours setup time or we say we failed to deliver it and just refund you.
Then, expect there to be non-early birds on them but maybe $5 a month higher after X quantities sold. MSRP starts at $49.
Notes:
Non-ECC memory, no IPMI. KVM will likely be optional and $15 or so setup fee and/or $1-2/mo. We can technically do ECC and IPMI, for +$40/mo but I doubt anyone would want that.
They'll likely come with a /30 IPv4 and /48 to /64 IPv6 (probably /56.) Higher amount available for IPv4 but not at a price that'd be attractive for the sale.
*These are the ones that may switch over to Ryzen 2600's in the future. Not retroactively, so if you buy a 5500 and keep it, then it'll be left alone. These boards technically do the best on 3000 series so we'll see how it goes and a third version for processors might be introduced later down the line, but it's going to be 5500 for the first 20 to 30 of them.
**Might end up being 4x1TB SSD, depends on how many of each we have available. I remember them as being mostly 500GB but all the drives I've pulled out of storage have been 1TB SSDs.
**Haven't tested the actual limits, likely capped if we want it not running into issues. I actually don't even remember how many SATA ports we have on the boards, I could check in 15 seconds and not write all this out but I won't.
Let me know if there's any feedback on how these should be modified, if pricing looks good, and so on. We're pretty flexible with them at this stage. All I know is that I tried to put in both NVMe and a RAID controller and while that's technically possible, it would need to be done later with some custom bracket, specific controllers, etc.
What about a $100-169/y dedicated server, as an extremely early bird offer, limited to 1 or 2 qty?
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
M y I ts hot ryzen here @VirMach
really want ECC for better production environment(mY biLLioN BuSIness),and also want IPMI/or other controller like PiKVM
I remember we have mentioned that,maybe give us some discount by pay the ONE-TIME fee?
what about /29(5ips)?
want to plug some HDD as well
oh wait,is that general offer?was though it is for old E3 customer
PiKVM would cost more than one-time setup for a dongle, but the dongle is going to a pretty old multi-port KVM unit with limited simultaneous seats and also old enough to where there's no HTML5. We can pretty much do anything... but it will add to the cost. So PiKVM or newer multi-port switch plus dongle would be an extra $20 ($15 versus $35 setup.)
ECC can technically be done if you downgrade from 5500 to 2600 or 3600 (if we get any) for an extra $10 a month or so. It's not something we'd likely be offering but if you end up being interested I can customize it, and you'll need to get the 64GB memory server because the ECC sticks are larger. Or with a board that supports IPMI, around +$40 a month as mentioned (or $400 one-time.)
But with all that said, it sounds like you just need an Epyc server. By then it'll be cheaper than whatever mess this would be... BUT, we could do it if you really want a Ryzen. I can even do a hacky way of getting ECC to work with 5000 series. But it's hacky.
Epyc server would end up being around $59 a month... BUT... it's limited on disk. Since it'll be uses U.2/Enterprise. So something like 1TB base, $10/mo or $50 setup per additional 1TB or so, super rough pricing on that.
I feel like the prices already include those discounts, without the $350 one-time fee. I guess it's not a 5600X, it's a 5500, and it's already past the amount of NVMe included for that $350 one-time upgrade.
Oh maybe I should have mentioned this last part:
Anyone who previously had a dedicated server with us and was impacted by a delay in it being switched to Ryzen and you wanted Ryzen, or you were negatively affected by the time we got Biloh'd or the time we got Ernie'd then we would offer you an equivalent Ryzen bare metal at the same price you were paying. Unfortunately, since I know someone may ask, this doesn't mean we can include ECC memory or KVM or IPMI or more IPv4 even though the original technically had those thing(s.) Those are all additional line items and weren't calculated in my ability to make that offer. Also putting another disclaimer, this doesn't apply to any dedicated servers we may have given out for free. Maybe. Let's say all of this requires my approval but I'll be nice.
It'd be $200-400 one-time setup fee or something like $15 a month. The IP addresses won't be cheap on dedicated servers. The lowest I could see a /29 being would be $8 per month additional. Depends on how many I set aside for this, but at the beginning we're assuming no one wants it. So if more people want it, price would come down.
People can send in their own drives and I'll install it without any additional setup fees (for members of this community, at least for these initial purchases) as long as you do not ever want it back, even if you cancel (I'll make an exception to this if you're sending us a $500 hard drive or something like that and get pre-approved for me to track it so we know it's yours.)
Extra drives, those can end up being pretty cheap. I have a lot of them, depends on what's left over. But probably... super quick estimate.
+$3/mo per 2-3TB HDD or 500GB SSD
+$5/mo per 1TB SSD
+$10/mo per 2TB SSD or 6-8TB HDD
If you had a server with us before, see above, it might end up being around $170/yr for the cheapest ones we sold in the past. Sadly your proposed pricing wouldn't even cover colocation costs and I'm not trying to lose even more money yet on this location, allow it to become profitable first and that sounds more like something we'd do if we have extra empty ones in the future.
I could definitely do $169/yr for first year only but I don't think you guys like those kind of deals. I'll see what's possible.