I think it's time to say goodbye for over 6 months the VM in Seattle wasn't working moved to LAX and still not working. I know @VirMach will be happy to see these BF special go but never thought I had to wait over 6 months to even have someone look into the issue. Ticket still opened for over a month with no response # 840866 . Only request would be if you can make it up for a day so that I can copy my data. This is so disappointing
I feel the same thing with LAZ2Z019, I have created a ticket, comments in this group including PM @virmach but I don't know what has happened until now the VPS is off and in the next few months the activation period will expire. there has been no continuation in the past 6 months.
@xyz said:
I think it's time to say goodbye for over 6 months the VM in Seattle wasn't working moved to LAX and still not working. I know @VirMach will be happy to see these BF special go but never thought I had to wait over 6 months to even have someone look into the issue. ............
@VirMach Can I get update regarding LAX1Z012 as it has been offline for ages. So far I have tried opening a ticket and waited so far but it has been down from October of '23 but there is no response from your end. FYI This VM was offline for long time in Seattle so it was moved here.
Okay general update/information for everyone above.
LAX2Z019 - This was scheduled to be dropped off yesterday after many delays, but the bad luck continues as the server completely died. It will be worked on again in about one week and the goal will shift to redeploying everyone on a different server and migrating data back if available.
DALZ004/DALZ007 - Many hours of remote hands wasn't able to get these in a functional state. These servers were previously functional so my assumption is that either remote hands in Atlanta is still not very helpful or that they may have been damaged during the delayed release/shipment process by Flex. I had plans to visit the facility about a month ago but that didn't pan out very well. Will likely be redeployed alongside LAX2Z019.
@simonindia said: @VirMach Can I get update regarding LAX1Z012 as it has been offline for ages. So far I have tried opening a ticket and waited so far but it has been down from October of '23 but there is no response from your end. FYI This VM was offline for long time in Seattle so it was moved here.
This shouldn't have been down since October 2023, it means your VM has other issues so I don't have a timeline for you specifically. Although the outage I have on file on my end is still a very long time at 47 days.
LAX1Z012 - Several attempts have been made to get this in a functional state. I actually don't have any further information on this one, other than resolution is still to be determined and we'll likely work in redeployments with LAX2Z019.
All of the above mentioned servers will hopefully be the last of their kind (indefinite outage) and to celebrate and apologize, they will come with an automatic and significant service expiration date extension. I can very unconfidently say that all of the above should be handled by approximately February 7th.
This will also likely happen as I push for the new credit system to be completed, so it may mean moving forward we'll go back to auto crediting/extending instead of manual mode. Once we do release that new system it'll come with an announcement on our site which will ask everyone to claim any outstanding credits we owe on previous outages by a certain date (probably several months from now will be the cutoff) with a pre-filled ticket form to keep things tidy.
ATLZ005 - Not many people on this one but this one has also been down for a while. It'll likely hitch a ride with the Dallas servers, but there's still some hope we can just get it back online to transfer services first.
Frankfurt - As of this moment the general plan is to close the location by May 1st. So the failing servers will probably be rescheduled to be moved away around the end of the week of February 1st. And the remaining two will get notices around late March/early April with the move happening sometime around mid-April (plans may change, this is early news.)
Other Plans - I also have a list of about a dozen other servers that have been naughty. We're going to (but with lower priority) try to also move services on these off to healthier nodes and have them all sent back to us so I can give them a spanking.
We are also going to try to send a "donor" server to all locations which will either be harvested for parts as needed or used as an immediate drop-in temporary replacement.
@cybertech said:
what happened to Tokyo in general? mine is still up for 177 days.
That is quite obvious - VirMach banned/closed the accounts or everyone that was actively using Tokyo and now only idlers!
then it should not require the upcoming maintenance.
I'm walking to Tokyo with thermal paste and a screwdriver, so we'll see what happens, but I opened the general notice for the time being. Once I'm there and have a more solid plan then some or all servers will be affected for some time and I'll try to provide more specific updates. A couple servers may be moved around, BIOS settings may be adjusted, a few ethernet and power cables may come loose, but hopefully no broken finger this time. The level of information provided before the maintenance will completely depend on the quality of the WiFi on the plane.
Tokyo - This location will be getting half a dozen additional servers. We may move some customers on nodes marked as potentially problematic so I can make improvements. TYOC037 and TYOC002 are getting new thermal paste. TYOC007, TYOC026, TYOC027, TYOC029, and TYOC039 are going to have some surgery performed on them, weather permitting.
If @Flying_Chinaman is still around then we might do a high RAM Epyc LES only Tokyo special in the comments over the next few days/weeks/months/year.
Any questions? I'll let @Jab answer for me as I liked his last few official responses.
Take some photos, irl photos, outside datacenter, you know... We know you are going there for "a job", but god damn it - use it also for some personal time, especially after such long flight.
@VirMach said: Any questions? I'll let @Jab answer for me as I liked his last few official responses.
TL;DR:
VirMach personally going to Tokyo to break all the nodes.
Frankfurt officially cancelled 1st of May.
New credit-for-outage system going live soon so everyone gonna be swim in credits... but only servers left will be Buffalo ColoCrossing.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
(added) Update 12:12AM Japan Standard Time - TYOC037 and TYOC002 will be taken offline soon for thermal paste re-application. They are being placed in maintenance mode and controls will be unavailable while we gracefully power down virtual servers to prepare.
(added)
(added) Update #2 - TYOC028 TYOC030 will be powered down for several minutes. Graceful per above.
(added)
(added) Update #3 - TYOC029, TYOC031, and TYOC033 may possibly lose connectivity briefly or be power cycled. This is not expected but may occur.
I have a feeling I should ramp up the number of checks with such quick updates
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Update January 25th, 8:45AM JST (Scheduled 10AM to 6PM.)
We are awaiting a delivery for TYOC002S and will proceed with a second powerdown and reboot once the required part (CPU heatsink) arrives.
All servers will briefly lose connectivity once parts arrive (by 12PM scheduled) to rearrange networking.
All servers will be power cycled for approximately 5 to 10 minutes as maintenance concludes at approximately 6PM.
Any other specific updates will be provided below.
@VirMach said: I hate that server so much after touching it yesterday.
>
Will the node unlock after you leave Tokyo? Some wrong configuration is keeping me out of the VM.
Message me information, it shouldn't be a problem on our end but I do apologize for leaving it on maintenance mode. Taken off now, will be further maintenance this evening if new heatsink arrives. I also got rid of some abusers for you.
@VirMach said: I hate that server so much after touching it yesterday.
>
Will the node unlock after you leave Tokyo? Some wrong configuration is keeping me out of the VM.
Message me information, it shouldn't be a problem on our end but I do apologize for leaving it on maintenance mode. Taken off now, will be further maintenance this evening if new heatsink arrives. I also got rid of some abusers for you.
All good now, just figured out that I typed the wrong SSH port. xD
Maybe you can relax in some hot spring resort after work is done.
@fan said: Maybe you can relax in some hot spring resort after work is done.
I'm a little overwhelmed visiting Tokyo for the first time so I think that might be more appropriate for a second more well-planned visit. So far I've gotten lost once for a few hours because I took the wrong phone (which didn't have cell service) and that was just trying to get lunch yesterday. I did get to see this cool statue though as a result.
@fan said: Maybe you can relax in some hot spring resort after work is done.
I'm a little overwhelmed visiting Tokyo for the first time so I think that might be more appropriate for a second more well-planned visit. So far I've gotten lost once for a few hours because I took the wrong phone (which didn't have cell service) and that was just trying to get lunch yesterday. I did get to see this cool statue though as a result.
@foitin said: 10Gbps port of Storage VPS issue not gonna be fixed/upgraded this time?
The 10Gbps on the storage idea was officially nixed at some point in the past with it officially switched over to 1Gbps on our website and anyone asking about it being told it was no longer planned & ask them if they'd like a pro-rated refund if they no longer wish to use the service given that fact.
It's still technically possible but I really really really really do not want to deal with an influx of abuse in Tokyo on a 10Gbps port. If we do it in the future (quietly) I'll reach out to you on LES and any other trusted users to switch over your VPS to 10Gbps or we'll go through everyone who was originally sold it as 10Gbps and verify that they do not have any potential issues with their account and quietly switch it over. That was supposed to happen on this trip but someone and I'm not pointing any fingers (me) forgot to bring the switch. Honestly though with everything else that could have gone wrong with this visit, I did pretty OK.
@foitin said: Gundam statute is only one station away from Tokyo Telecom Center where Equinix TY 11 is in. But I remember virmachs not in TY11?
Hotel is strategically halfway between airport (Haneda) and the datacenter. I was basically walking in big circles in Koto area (I think, right?) trying to find this specific restaurant that opened at 11AM and then eventually around 2PM gave up after I realized the went out of business and ate at the food court next to the statue.
@cybertech said:
so what new disks are put into Tokyo servers?
Kioxa, PNY Gen4, SKHynix Gold, Samsung Pro so far. Saving the Mushkin and any others for the less critical additional disks for nodes that don't really need it.
this specific restaurant that opened at 11AM and then eventually around 2PM gave up after I realized the went out of business
You cant possibly come to Japan only eat panda express, can you? I mean who would do that :'(
Daiba is not where you go for food anyway. It's for young couples datings and stuff.
this specific restaurant that opened at 11AM and then eventually around 2PM gave up after I realized the went out of business
You cant possibly come to Japan only eat panda express, can you? I mean who would do that :'(
Daiba is not where you go for food anyway. It's for young couples datings and stuff.
actually everything can be settled through UberEats.
Amazon dropped the ball on delivery to tidy up all the cables & change heatsink for TYOC002S. First it was January 24th, then 25th morning, but I got them to try to squeeze it into a 10PM delivery tonight so I'll revisit again if they make it on time (and try again for tomorrow morning.)
this specific restaurant that opened at 11AM and then eventually around 2PM gave up after I realized the went out of business
You cant possibly come to Japan only eat panda express, can you? I mean who would do that :'(
Daiba is not where you go for food anyway. It's for young couples datings and stuff.
I hadn't eaten or slept since I landed so I just gave up and ate there after getting lost. Feel free to recommend any restaurant for today but remember we're working on a low budget no A5 Kobe
Have Amazon ship to Lawson/Family Mart and theyll only use legit courier Yamato/Japan Post. They deliver on time and you can pick it any time after arrival.
10PM delivery Another poor postman gotta work at night without overtime pay from Amazon.
For restaurante, corny ramen chain like Ichiran? Probably the least disliked food.
Never heard of anyone hated it unlike other famous chains where people either like it or hate it.
@foitin said: Have Amazon ship to Lawson/Family Mart and theyll only use legit courier Yamato/Japan Post. They deliver on time and you can pick it any time after arrival.
Yeah the Yamato delivered item arrived today but sadly that was the least important item
@foitin said: 10PM delivery Another poor postman gotta work at night without overtime pay from Amazon.
10PM to 2AM was the timeslot I think but maybe they'll delay it again because now it says "Problem" and "check back January 27th"
@foitin said: For restaurante, corny ramen chain like Ichiran? Probably the least disliked food.
I'll see if that's closer or Tsujita as they had one in Los Angeles and it's probably not considered that great here since it's a chain but in LA it's the best available.
Edit -- Leaning toward Tsujita as it seems unless Google Maps is being weird, Ichiran is more north and out the way plus I like tsukemen better
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Comments
That is quite obvious - VirMach banned/closed the accounts or everyone that was actively using Tokyo and now only idlers!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Too quiet. There would have been a lot more squawking if that were the case.
then it should not require the upcoming maintenance.
I bench YABS 24/7/365 unless it's a leap year.
Maintenance is there to consolidate servers and pay even less! Same amount of idlers less power draw. You will see!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
oh! that makes sense now.
I bench YABS 24/7/365 unless it's a leap year.
Anyone on LAXA031.VIRM? my VM went down 13 hours ago but the status page shows everything is up and working.
Same also on LAXA031 and VPS down.
I am on this one. My VM is currently working fine, but it looks like the control panel gives a timeout. Maybe a disk affecting some VMs?
I think it's time to say goodbye for over 6 months the VM in Seattle wasn't working moved to LAX and still not working. I know @VirMach will be happy to see these BF special go but never thought I had to wait over 6 months to even have someone look into the issue. Ticket still opened for over a month with no response # 840866 . Only request would be if you can make it up for a day so that I can copy my data. This is so disappointing
I feel the same thing with LAZ2Z019, I have created a ticket, comments in this group including PM @virmach but I don't know what has happened until now the VPS is off and in the next few months the activation period will expire. there has been no continuation in the past 6 months.
@VirMach Can I get update regarding LAX1Z012 as it has been offline for ages. So far I have tried opening a ticket and waited so far but it has been down from October of '23 but there is no response from your end. FYI This VM was offline for long time in Seattle so it was moved here.
I thought LAX1Z012 got fixed. What're you seeing?
Okay general update/information for everyone above.
LAX2Z019 - This was scheduled to be dropped off yesterday after many delays, but the bad luck continues as the server completely died. It will be worked on again in about one week and the goal will shift to redeploying everyone on a different server and migrating data back if available.
DALZ004/DALZ007 - Many hours of remote hands wasn't able to get these in a functional state. These servers were previously functional so my assumption is that either remote hands in Atlanta is still not very helpful or that they may have been damaged during the delayed release/shipment process by Flex. I had plans to visit the facility about a month ago but that didn't pan out very well. Will likely be redeployed alongside LAX2Z019.
This shouldn't have been down since October 2023, it means your VM has other issues so I don't have a timeline for you specifically. Although the outage I have on file on my end is still a very long time at 47 days.
LAX1Z012 - Several attempts have been made to get this in a functional state. I actually don't have any further information on this one, other than resolution is still to be determined and we'll likely work in redeployments with LAX2Z019.
All of the above mentioned servers will hopefully be the last of their kind (indefinite outage) and to celebrate and apologize, they will come with an automatic and significant service expiration date extension. I can very unconfidently say that all of the above should be handled by approximately February 7th.
This will also likely happen as I push for the new credit system to be completed, so it may mean moving forward we'll go back to auto crediting/extending instead of manual mode. Once we do release that new system it'll come with an announcement on our site which will ask everyone to claim any outstanding credits we owe on previous outages by a certain date (probably several months from now will be the cutoff) with a pre-filled ticket form to keep things tidy.
ATLZ005 - Not many people on this one but this one has also been down for a while. It'll likely hitch a ride with the Dallas servers, but there's still some hope we can just get it back online to transfer services first.
Frankfurt - As of this moment the general plan is to close the location by May 1st. So the failing servers will probably be rescheduled to be moved away around the end of the week of February 1st. And the remaining two will get notices around late March/early April with the move happening sometime around mid-April (plans may change, this is early news.)
Other Plans - I also have a list of about a dozen other servers that have been naughty. We're going to (but with lower priority) try to also move services on these off to healthier nodes and have them all sent back to us so I can give them a spanking.
We are also going to try to send a "donor" server to all locations which will either be harvested for parts as needed or used as an immediate drop-in temporary replacement.
I'm walking to Tokyo with thermal paste and a screwdriver, so we'll see what happens, but I opened the general notice for the time being. Once I'm there and have a more solid plan then some or all servers will be affected for some time and I'll try to provide more specific updates. A couple servers may be moved around, BIOS settings may be adjusted, a few ethernet and power cables may come loose, but hopefully no broken finger this time. The level of information provided before the maintenance will completely depend on the quality of the WiFi on the plane.
Tokyo - This location will be getting half a dozen additional servers. We may move some customers on nodes marked as potentially problematic so I can make improvements. TYOC037 and TYOC002 are getting new thermal paste. TYOC007, TYOC026, TYOC027, TYOC029, and TYOC039 are going to have some surgery performed on them, weather permitting.
If @Flying_Chinaman is still around then we might do a high RAM Epyc LES only Tokyo special in the comments over the next few days/weeks/months/year.
Any questions? I'll let @Jab answer for me as I liked his last few official responses.
Have fucking fun in Tokyo!
Take some photos, irl photos, outside datacenter, you know... We know you are going there for "a job", but god damn it - use it also for some personal time, especially after such long flight.
TL;DR:
VirMach personally going to Tokyo to break all the nodes.
Frankfurt officially cancelled 1st of May.
New credit-for-outage system going live soon so everyone gonna be swim in credits... but only servers left will be Buffalo ColoCrossing.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
due date is Feb. 4th, so no luck for credit before 7th. too bad.
So you're going to let it go?
I let many go by and this one will go too. pity.
how about Amsterdam? what happened to the migration (AMS to AMS)
I bench YABS 24/7/365 unless it's a leap year.
It seems that the trumpet of victory is heard.
Same thing happens again!
@FrankZ goes away, @VirMach "flies" to Tokyo.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Looks like maintenance is ongoing in Tokyo, RIP the uptime of the storage box.
I have a feeling I should ramp up the number of checks with such quick updates
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
I hate that server so much after touching it yesterday.
Story time or still too much hatred?
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
TYOC037 all good after reboot!
I bench YABS 24/7/365 unless it's a leap year.
>
Will the node unlock after you leave Tokyo? Some wrong configuration is keeping me out of the VM.
Update January 25th, 8:45AM JST (Scheduled 10AM to 6PM.)
I hope everything goes well.
is virmach flying to Netherlands after?
I bench YABS 24/7/365 unless it's a leap year.
Message me information, it shouldn't be a problem on our end but I do apologize for leaving it on maintenance mode. Taken off now, will be further maintenance this evening if new heatsink arrives. I also got rid of some abusers for you.
Frankfurt then Amsterdam.
All good now, just figured out that I typed the wrong SSH port. xD
Maybe you can relax in some hot spring resort after work is done.
your life my dream
I bench YABS 24/7/365 unless it's a leap year.
I'm a little overwhelmed visiting Tokyo for the first time so I think that might be more appropriate for a second more well-planned visit. So far I've gotten lost once for a few hours because I took the wrong phone (which didn't have cell service) and that was just trying to get lunch yesterday. I did get to see this cool statue though as a result.
that's awesome
George Datacenter LLC
www.georgedatacenter.com
Owner Hardware
so what new disks are put into Tokyo servers?
I bench YABS 24/7/365 unless it's a leap year.
floppy disks
and Tokyo IPv6, 10Gbps port of Storage VPS issue not gonna be fixed/upgraded this time?
Gundam statute is only one station away from Tokyo Telecom Center where Equinix TY 11 is in. But I remember virmachs not in TY11?
The 10Gbps on the storage idea was officially nixed at some point in the past with it officially switched over to 1Gbps on our website and anyone asking about it being told it was no longer planned & ask them if they'd like a pro-rated refund if they no longer wish to use the service given that fact.
It's still technically possible but I really really really really do not want to deal with an influx of abuse in Tokyo on a 10Gbps port. If we do it in the future (quietly) I'll reach out to you on LES and any other trusted users to switch over your VPS to 10Gbps or we'll go through everyone who was originally sold it as 10Gbps and verify that they do not have any potential issues with their account and quietly switch it over. That was supposed to happen on this trip but someone and I'm not pointing any fingers (me) forgot to bring the switch. Honestly though with everything else that could have gone wrong with this visit, I did pretty OK.
Hotel is strategically halfway between airport (Haneda) and the datacenter. I was basically walking in big circles in Koto area (I think, right?) trying to find this specific restaurant that opened at 11AM and then eventually around 2PM gave up after I realized the went out of business and ate at the food court next to the statue.
Kioxa, PNY Gen4, SKHynix Gold, Samsung Pro so far. Saving the Mushkin and any others for the less critical additional disks for nodes that don't really need it.
now we talkin
I bench YABS 24/7/365 unless it's a leap year.
You cant possibly come to Japan only eat panda express, can you? I mean who would do that :'(
Daiba is not where you go for food anyway. It's for young couples datings and stuff.
actually everything can be settled through UberEats.
I bench YABS 24/7/365 unless it's a leap year.
Amazon dropped the ball on delivery to tidy up all the cables & change heatsink for TYOC002S. First it was January 24th, then 25th morning, but I got them to try to squeeze it into a 10PM delivery tonight so I'll revisit again if they make it on time (and try again for tomorrow morning.)
I hadn't eaten or slept since I landed so I just gave up and ate there after getting lost. Feel free to recommend any restaurant for today but remember we're working on a low budget
no A5 Kobe
Have Amazon ship to Lawson/Family Mart and theyll only use legit courier Yamato/Japan Post. They deliver on time and you can pick it any time after arrival.
Another poor postman gotta work at night without overtime pay from Amazon.
10PM delivery
For restaurante, corny ramen chain like Ichiran? Probably the least disliked food.
Never heard of anyone hated it unlike other famous chains where people either like it or hate it.
Yeah the Yamato delivered item arrived today but sadly that was the least important item
10PM to 2AM was the timeslot I think
but maybe they'll delay it again because now it says "Problem" and "check back January 27th"
I'll see if that's closer or Tsujita as they had one in Los Angeles and it's probably not considered that great here since it's a chain but in LA it's the best available.
Edit -- Leaning toward Tsujita as it seems unless Google Maps is being weird, Ichiran is more north and out the way plus I like tsukemen better
Oh, is VirMach in Japan today? If you get hungry at night, I recommend going to 7-Eleven and buying Tomita ramen.
https://www.sej.co.jp/products/a/item/091063/
dnscry.pt - Public DNSCrypt resolvers hosted by LowEnd providers • Need a free NAT LXC? -> https://microlxc.net/
Despite my VM is going to be moved to NL, mine is back from dead and still in xtom infra.
https://microlxc.net/
NL is xtom
Ramen - yuk! The texture of animal fat. :barf:
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Walking?
The terrain in western Alaska is rough.
We'd recommend a sailboat.
It's faster than walking, without increasing carbon footprint.
Screwdriver?
We'd recommend a wirecutter and a hammer.
It's more effective in ensuring the servers stay offline for as long as possible.
We accept Karma donations for the last flan. 🍮 affbrr