all my 47.87.* are unreachable, although some are active on the panel. So must be like what Frank said about a drunk lonely - pizza eating - lonely - frustrated minion.
My current services chart. The web servers are the vertical axis, and the DNS/monitoring VMs are along the bottom. The yellow N/A's are non reporting DNS/monitoring VMs. The red "Down" is non responding web servers. All of these N/A's and Down's are on VirMach 47.87.xx.xx.
This is the benefit of running a redundant network of LE VMs. HetrixTools has not reported any of my services or web sites as down.
I'm not sure. Firstly I checked the 'Network Status' page on their website, and they not mentioned any network maintenance. After asked it on the forum.
Some of these nodes are abnormal:
Atlanta ATLZ007
Atlanta ATLZ010
Tampa TPAZ002
Tampa TPAZ003
Dallas DALZ003
Dallas DALZ005
Miami MIAZ012
Miami MIAZ014
NewYorkCity NYCB013
NewYorkCity NYCB046
LosAngeles LAXA006
Chicago CHI2Z026
:'(
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural. It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@flips said:
Seems both Hetrix and BetterUptime doesn't receive any heartbeat from my VM's in Amsterdam and Frankfurt ...
I can reach my Amsterdam Virmach VPS from several locations, but not from home (through one of the largest dutch internet providers) or a few other of my VPS servers. Already goes wrong from the first or second hop. The AMS-IX seems to know the route, however, other places (earlier hops) do not.
When I look at several of the users on the Virmach VPS, there are quite a few different providers here in NL that still have a possibility to connect, but as said, not everybody. My guess it's a BGP/Routing issue or something with a broken interconnect
@lesuser said:
It is reassuring to hear that there has been no update from VirMach after so many hours.
Was sleeping, I assumed due to the number of outages it was likely the monitoring service going down/having issues since it was across multiple datacenters/data carriers. Right now it looks like just Amsterdam but depending on where I look it varies. Doing a more extensive check now.
(edit) Okay more are appearing down again on monitoring randomly.
@VirMach it seems like everything 47.87.x.x is basically dead, with a few people/routes still working just barely (maybe old info left in those routers)
@Daevien said: @VirMach it seems like everything 47.87.x.x is basically dead, with a few people/routes still working just barely (maybe old info left in those routers)
Yeah, looks like it's the 47.87.128.0/17 block.
Theory #1: they changed their bank details and when sending the payment, without our knowledge it was delayed several days by our bank, it's still "pending review" by the bank. No notification from the bank or them, or any datacenter regarding a de-announcement request though and it still appears to be on our ARIN.
Could possibly also be unrelated and something else related to that. I'm contacting everyone now to see what's going on and also checking the routes to see how it's flapping. Contacting bank in the morning, already sent off something to the IP subnet provider. Reaching out/checking for communication from DCs now.
@Daevien said: @VirMach it seems like everything 47.87.x.x is basically dead, with a few people/routes still working just barely (maybe old info left in those routers)
Yeah, looks like it's the 47.87.128.0/17 block.
Theory #1: they changed their bank details and when sending the payment, without our knowledge it was delayed several days by our bank, it's still "pending review" by the bank. No notification from the bank or them, or any datacenter regarding a de-announcement request though and it still appears to be on our ARIN.
Could possibly also be unrelated and something else related to that. I'm contacting everyone now to see what's going on and also checking the routes to see how it's flapping. Contacting bank in the morning, already sent off something to the IP subnet provider. Reaching out/checking for communication from DCs now.
hello, man, u still alive?
i hope you are well and family is good.
you should be more active, at least in LES, we all have our pants down so you can feel at home
ehab
ps: edited for my finglish before @AlwaysSkint fingers me again.
I hope it can be resolved soon. Calling the bank as soon as they open or driving down there, still awaiting response by them, looks like records were modified and it's conflicting which explains how the issues transpired.
We're ready to do emergency switches if necessary, we can get blocks from IPXO, but best to avoid that as it may cause another ticket backlog. They're obviously building up already.
In any case it unfortunately does look like some issues will continue until everything updated back. It sure would be nice to own our own IPv4 blocks. I just need to take out a small loan of 2 million dollars for that. We might buy one or two /24 and sell it as an optional $50 one-time fee for a true static 1x IPv4 for those that need it to avoid future musical chairs with IPv4. All 2 customers that make that purchase will appreciate it.
https://billing.virmach.com/networkissuesrss.php had a change.
---
(changed) <![CDATA[VirMach® Network Issues Feed]]> https://billing.virmach.com/networkissues.php https://billing.virmach.com/networkissues.php?view=nid1488 Mon, 27 Mar 2023 07:39:00 -0700 <![CDATA[
(into ) <![CDATA[VirMach® Network Issues Feed]]> https://billing.virmach.com/networkissues.php https://billing.virmach.com/networkissues.php?view=nid1496 Wed, 05 Apr 2023 01:30:00 -0700 <![CDATA[
(into )
(into ) This appears to be a network outage. We'll update you with any details.
(into )
(into ) Update: this appears to be an issue with the IPv4 block 47.87.128.0/17
(into )
(into ) ]]> https://billing.virmach.com/networkissues.php?view=nid1488 Mon, 27 Mar 2023 07:39:00 -0700 <
Comments
Do their ranges still have proper IRR route records?
dnscry.pt - Public DNSCrypt resolvers hosted by LowEnd providers • Need a free NAT LXC? -> https://microlxc.net/
Status page is showing many services down
https://statusbot.virm.ac/en/
Powerful AMD Ryzen VPS (aff)
all my 47.87.* are unreachable, although some are active on the panel. So must be like what Frank said about a drunk lonely - pizza eating - lonely - frustrated minion.
My current services chart. The web servers are the vertical axis, and the DNS/monitoring VMs are along the bottom. The yellow N/A's are non reporting DNS/monitoring VMs. The red "Down" is non responding web servers. All of these N/A's and Down's are on VirMach 47.87.xx.xx.
This is the benefit of running a redundant network of LE VMs. HetrixTools has not reported any of my services or web sites as down.
I'm not sure. Firstly I checked the 'Network Status' page on their website, and they not mentioned any network maintenance. After asked it on the forum.
My other VM's in USA looks like also affected
LAXA026 - 47.87.153.XXX - https://i.imgur.com/OszlBVp.png
DALZ003 - 47.87.196.XXX - https://i.imgur.com/thixxNT.png
SEAZ010 - 47.87.192.XXX - https://i.imgur.com/L7uPkf2.png
Some of them accessible from my local (home) ISP, some of them not.
Some of these nodes are abnormal:
Atlanta ATLZ007
Atlanta ATLZ010
Tampa TPAZ002
Tampa TPAZ003
Dallas DALZ003
Dallas DALZ005
Miami MIAZ012
Miami MIAZ014
NewYorkCity NYCB013
NewYorkCity NYCB046
LosAngeles LAXA006
Chicago CHI2Z026
:'(
47.87.* down ?
weird. i can access internally SSH / webpanel but it cannot load my website.
I bench YABS 24/7/365 unless it's a leap year.
Can you tell me ?
https://bgp.tools/prefix/47.87.224.0/24#validation
https://bgp.tools/prefix/47.87.201.0/24#validation
not entirely, i can reach 2 vps in LA but they cannot reach any ubuntu update servers.
some network mess that affects mutliple locations in US and EU.
Has there been any announcement by VirMach?
Powerful AMD Ryzen VPS (aff)
Seems both Hetrix and BetterUptime doesn't receive any heartbeat from my VM's in Amsterdam and Frankfurt ...
I hate waking up to 102 Hetrix emails! Grrr.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Any update from VirMach?
Powerful AMD Ryzen VPS (aff)
Do you all have IPv6? My ty021 lax02 currently does not have an ipv6, and it has been 2 months since I applied for it
Hit snooze and you can wake up to more?
Nah. I got it by default with one of my latest vpses but this is it. Other my boxes are without.
I can reach my Amsterdam Virmach VPS from several locations, but not from home (through one of the largest dutch internet providers) or a few other of my VPS servers. Already goes wrong from the first or second hop. The AMS-IX seems to know the route, however, other places (earlier hops) do not.
When I look at several of the users on the Virmach VPS, there are quite a few different providers here in NL that still have a possibility to connect, but as said, not everybody. My guess it's a BGP/Routing issue or something with a broken interconnect
Not yet, but I expect they are working on figuring out the next step..
LAXA024It is reassuring to hear that there has been no update from VirMach after so many hours.
Powerful AMD Ryzen VPS (aff)
yes.
no problem
nooo probblemo
Was sleeping, I assumed due to the number of outages it was likely the monitoring service going down/having issues since it was across multiple datacenters/data carriers. Right now it looks like just Amsterdam but depending on where I look it varies. Doing a more extensive check now.
(edit) Okay more are appearing down again on monitoring randomly.
@VirMach it seems like everything 47.87.x.x is basically dead, with a few people/routes still working just barely (maybe old info left in those routers)
Yeah, looks like it's the 47.87.128.0/17 block.
Theory #1: they changed their bank details and when sending the payment, without our knowledge it was delayed several days by our bank, it's still "pending review" by the bank. No notification from the bank or them, or any datacenter regarding a de-announcement request though and it still appears to be on our ARIN.
Could possibly also be unrelated and something else related to that. I'm contacting everyone now to see what's going on and also checking the routes to see how it's flapping. Contacting bank in the morning, already sent off something to the IP subnet provider. Reaching out/checking for communication from DCs now.
Thanks
hello, man, u still alive?
i hope you are well and family is good.
you should be more active, at least in LES, we all have our pants down so you can feel at home
ehab
ps: edited for my finglish before @AlwaysSkint fingers me again.
Always a pleasure to see you @netomx
I hope it can be resolved soon. Calling the bank as soon as they open or driving down there, still awaiting response by them, looks like records were modified and it's conflicting which explains how the issues transpired.
We're ready to do emergency switches if necessary, we can get blocks from IPXO, but best to avoid that as it may cause another ticket backlog. They're obviously building up already.
In any case it unfortunately does look like some issues will continue until everything updated back. It sure would be nice to own our own IPv4 blocks. I just need to take out a small loan of 2 million dollars for that. We might buy one or two /24 and sell it as an optional $50 one-time fee for a true static 1x IPv4 for those that need it to avoid future musical chairs with IPv4. All 2 customers that make that purchase will appreciate it.
Fucking Alibaba will make HetrixTools bankrupt with that kind of rate.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
virmach has just been alibaba-ed.
I bench YABS 24/7/365 unless it's a leap year.
Non Alibaba IP still alive and healthy btw.
https://microlxc.net/
I'm just very grateful that Tokyo isn't Alibaba'd or else we'd have 3,000 tickets right now.
You waited like a month to re-post this... and you using it wrong? Why would you like HT to bankrupt?!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
One day Alibaba gives rDNS, next day Alibaba takes everything.
China! Blame China!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
LOL
the best post i saw from you is the fish image.
when ever i see you post .. it itches not to repost it
sorry
will do again.
how are you now?
Wait, I can start ticket for each of my affected services and get SLA credits - grand total of 0.02$?!
Mmmmmm
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Okey, the actual fuck is this. We sure this is Alibaba related - like if this is payment/bank issue it would be deleted from BGP so it should timeout from everything, not only part (?)
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
Welcome to the internet. This is one reason one of my friends constantly complains about BGP. It's why AWS can have widespread outages from one Amazon engineer fat-fingering something.
I was just typing that it'd be likely that other sites/connections would also "timeout", so I already took the affected VPS offline from my side earlier this morning to let the connections gradually move to other ones, and just when I'm typing this I get the message from 2 other monitoring points I have that it's not longer available. So that is confirmation.
Could come back pretty quickly when things are sorted out though...
Make that 3 by counting me in
Amsterdam and Frankfurt will receive IPv4 changes throughout the next several hours. Old IP will remain there most likely, plus new. We're sending out mass email now and then most likely you'll receive a ticket with the change. These will be blocks from Tokyo moving over so geolocation may be incorrect for some time and I don't know exact timeline of the change propagating everywhere.
I'm very very rusty on the scripts we use for these changes, so the level of notice we're able to provide depends on that, as the person usually handling these isn't here at 4AM.
Next up may end up being emergency migrations from HV over to nodes that have non-47.87.xx.xx blocks since we have to eventually do that anyway. I figure since it's going to be spotty/no connectivity for some time it won't hurt to do these now.
This means my previous message saying we're sending 10 to Ryzen plus 1 Epyc will just be cut down to 1 Epyc only, for some time, as I assume if we get the 47.87.xx.xx block back there will be a lot of complaints from people receiving IPv4 from that block in Tokyo.
Some hour ago died also my .de VPS from Tomazu/Webhosting24 and day isn't over yet
Have you noticed that TYO007 is offline? Eight hours have passed and no exceptions have been reported in the network status.
Any time someone reports a Tokyo outage, I am very skeptical until it is proven. Because I would accurately say 99% of the time it is an inaccurate report. However in this case I already saw that, and I'm aware. Our monitoring was showing it online but enough reports and also other report from 2nd monitoring system. I'm dealing with a larger emergency at the moment. The server is definitely facing an issue. It's partially online but effectively offline. I'll open a network status with more information as soon as possible.
Okay, thanks for your reply.
@VirMach unfortunately your billing website is dead now.
https://microlxc.net/