VirMach - Complain - Moan - Praise - Chit Chat - Flan

1213214216218219274

Comments

  • FrankZFrankZ ModeratorOG

    @vyas said:
    @FrankZ convinced Virmach to make all the changes before April

    Vacation time comes

  • Wondering if the NYC work may have borked some network configuration. NYCB018 isn't in the list of hosts on the network status page, but networking is down. VNC appears to be working. The gateway at 141.11.235.1 appears down.

  • FrankZFrankZ ModeratorOG

    @Brayvin said:
    Wondering if the NYC work may have borked some network configuration. NYCB018 isn't in the list of hosts on the network status page, but networking is down. VNC appears to be working. The gateway at 141.11.235.1 appears down.

    141.11.235.1 looks up to me. Just saying.

     4  ae-2.r22.nwrknj03.us.bb.gin.ntt.net (129.250.3.174)  3.512 ms  3.490 ms  3.462 ms
     5  ae-0.a01.nwrknj03.us.bb.gin.ntt.net (129.250.2.97)  3.530 ms  3.510 ms  3.516 ms
     6  e0-16.cr1.nyc2.us.unitasglobal.net (50.115.91.10)  4.064 ms  5.070 ms  5.275 ms
     7  e0-40.cr1.nyc1.us.unitasglobal.net (50.115.90.134)  4.829 ms * *
     8  inap.cust.nyc1.us.unitasglobal.net (198.232.115.146)  3.642 ms  3.780 ms  3.744 ms
     9  border1-po2-bbnet2.nyj004.us.unitasglobal.net (216.52.95.109)  4.442 ms border1-po1-bbnet1.nyj004.us.unitasglobal.net (216.52.95.46)  4.667 ms  4.639 ms
    10  74.201.136.182 (74.201.136.182)  4.517 ms  4.397 ms  4.364 ms
    11  hosted-by.royalehosting.net (45.59.169.177)  23.207 ms  23.124 ms  83.339 ms
    12  141.11.235.1 (141.11.235.1)  5.974 ms  5.930 ms  5.891 ms
    
  • @FrankZ said:

    @Brayvin said:
    Wondering if the NYC work may have borked some network configuration. NYCB018 isn't in the list of hosts on the network status page, but networking is down. VNC appears to be working. The gateway at 141.11.235.1 appears down.

    141.11.235.1 looks up to me. Just saying.

    Yes, I neglected to mention that it seems down from the VPS point of view.

  • edited March 2025

    @Brayvin said:

    @FrankZ said:

    @Brayvin said:
    Wondering if the NYC work may have borked some network configuration. NYCB018 isn't in the list of hosts on the network status page, but networking is down. VNC appears to be working. The gateway at 141.11.235.1 appears down.

    141.11.235.1 looks up to me. Just saying.


    Yes, I neglected to mention that it seems down from the VPS point of view.

    Don't be afraid to make a ticket. We usually say to please not make one but this time around I went out of my way to insist on tickets being made and it was with good reason -- this was a huge mess to complete.

    I've already located your issue by looking at some data reports, one moment.

    (edit) Yours was one of these:

    A few services ran into problems (approximately 12~) and need to be migrated again or moved to another server due to high IPv4 requirements

    I made space for you to stay as you didn't have a high quantity of IPs and I wanted to avoid a second migration where possible, but then forgot to take you off the list to move, so when it was moved, the IP wasn't changed because it didn't match the subnet being processed for all the moves. Sorry about that, this took much longer than expected due to issues with the control panel and other (expected) complications and this was at the very end so I'm not firing on all cylinders right now. It's on NYCB018, with the right IP now.

  • It appears that the New York migration has resulted in total data loss.

    Booting from Hard Disk...
    Boot failed: not a bootable disk
    
    No bootable device.
    

    GParted has detected a 15GB block device without any partition:

    SolusVM says the server should have 60GB storage, not 15GB.
    We are losing millions per hour.

    Ticket Created #122818

    We accept Karma donations for the last flan. 🍮 affbrr

  • AlwaysSkintAlwaysSkint OG RETIRED

    @yoursunny said: We are losing millions per hour.

    You, as in I singular, may be but is anyone else? Plonker!

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • Chicago down and gone from the route map (at least my subnet) for 2 hours.

    Let's hope it's not quadranet gone... again.

  • msattmsatt ProviderOG

    @Kris said:
    Chicago down and gone from the route map (at least my subnet) for 2 hours.

    Let's hope it's not quadranet gone... again.

    Confirmed CHIZ029 and CHIZ030 both nodes offline with no CP access.

    Get your FREE VPS if you develop Open Source software

  • vyasvyas OG
    edited March 2025

    @AlwaysSkint said:

    @yoursunny said: We are losing millions per hour.

    You, as in I singular, may be but is anyone else? Plonker!

    First person plural indicating Victorian era (time traveller?) or Royalty (more likely given King of Push ups)

  • AlwaysSkintAlwaysSkint OG RETIRED

    @vyas said: First person plural..

    Yup, thus sticking with my assessment. :)

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • AlwaysSkintAlwaysSkint OG RETIRED
    edited March 2025

    CHIZ001 & CHIZ035 also down. Looks like Chicago is a dead zone: confirmed by https://status.virm.ac/. :(

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • JabJab TOP Member 2027
    edited March 2025

    We are aware of the outage in Chicago and are evaluating most recent backups and available space in NYC to load from backups.

    P A N I C

    Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.

  • FrankZFrankZ ModeratorOG

    Why do I expect something like this is running through VirMach's mind regarding Quadranet.

  • @Jab said: We are aware of the outage in Chicago and are evaluating most recent backups and available space in NYC to load from backups.

    Jesus christ.

  • AlwaysSkintAlwaysSkint OG RETIRED

    Was the writing not on the wall, for leaving services in Chicago, as is? Head-in-the-sand? More attention paid there, rather than the MJJ fest in Tokyo, might have been prudent?
    Shame to be moved to NYC, where I already have a surplus of services; San Jose would be a marginal improvement, since the demise of Seattle. Need more options in NW and central USA, IMHO.

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • QN said to move by end of April. I'm not going to pretend like I wasn't expecting this as a possibility, that's why we already had weekly backups off-site, but it would have been nice if we at least made it to the end of March.

    Most backups look like they're from March 21st. Backups aren't perfect, CHIZ001 is the most likely to potentially have some issues.

    I do not know why they're down, outside of the simple reason being "QuadraNet" so it could be they didn't pay transit, they didn't pay the network engineer, they decided to randomly move us to St Louis even though we didn't ask for it, or whatever else. NYC is low on available space after the migrations that happened this week. ATL has been low as well since Miami got moved into it, still working out a plan. If you need to be moved sooner, open a ticket.

    Miami, we did get a notification from QN that it's been given to Hivelocity. We are likely not continuing with Hivelocity since it is Hivelocity, but I'll let you guys know later today/tomorrow (for the remaining few customers who wanted to stay.)

  • @AlwaysSkint said:
    Was the writing not on the wall, for leaving services in Chicago, as is? Head-in-the-sand? More attention paid there, rather than the MJJ fest in Tokyo, might have been prudent?

    Everything was already assessed and this was the best way to do it. We can go over it later in a long post if anyone wants, just not now.

    @AlwaysSkint said: Need more options in NW and central USA, IMHO.

    I've been meaning to post about this, if I can just have 3 hours of breathing room this week :)

  • AlwaysSkintAlwaysSkint OG RETIRED

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • JabJab TOP Member 2027

    Update 8AM CST - This is related to networking. We are exploring other solutions, but the primary plan is still to perform emergency loading from backups. We will likely need to move some customers to Atlanta. If you have other location preferences, please open a ticket using the "Network Status" button.

    Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.

  • msattmsatt ProviderOG

    Ticket #859916
    The migration of your servers from Chicago to Amsterdam is scheduled to begin around March 21st. We'll confirm the exact timeline and any expected downtime soon. Please back up your data to prevent any potential data loss..........

    AI must have planned this ;-)

    Get your FREE VPS if you develop Open Source software

  • AlwaysSkintAlwaysSkint OG RETIRED
    edited March 2025

    Trying to open a Ticket using Network Status button, as requested - getting nowhere:
    Dialogue box shows the IP and a Close button, with..

    If you cannot make a ticket due to troubleshooter timing out, go back and click Network Status.

    Getting too stressed trying everything out and going in circles... leaving for now.

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • msattmsatt ProviderOG

    @AlwaysSkint said:
    Trying to open a Ticket using Network Status button, as requested - getting nowhere:
    Dialogue box shows the IP and a Close button, with..

    If you cannot make a ticket due to troubleshooter timing out, go back and click Network Status.

    Getting too stressed trying everything out and going in circles... leaving for now.

    Open a ticket | Contact us | Select VM | Network status
    Agree it is convoluted.

    Get your FREE VPS if you develop Open Source software

  • edited March 2025

    @AlwaysSkint said:
    Trying to open a Ticket using Network Status button, as requested - getting nowhere:
    Dialogue box shows the IP and a Close button, with..

    If you cannot make a ticket due to troubleshooter timing out, go back and click Network Status.

    Getting too stressed trying everything out and going in circles... leaving for now.

    We probably need a better name for them than buttons or departments. I'm going to do a quick check logged in as a customer just to communicate what I'm specifically verifying right now.

    • Service page takes a while to load, but does load and does load the cached server information as expected. This is the page where controls usually would exist and referred to as service details page.
    • Running connection troubleshooter (the simple version) on this page confirms it's offline and doesn't error out.
    • Contact support button on this page that shows up is clicked, takes us to ticket page
    • "Contact us" button is clickable. It populates list of services. I specifically select a service that's in Chicago.

    I'll share a screenshot at this point. Network status button is clickable, and I click continue.

    Then it takes me to the page to open a ticket. Let me know at which one of these steps you're having an issue on so I can check it further or if we should make any specific improvements to the process.

    (edit) Ohhh I think I have a guess, is it that the select service box looks like it already selected the service if it highlighted it? Or does the continue button actually just not appear?

    If you cannot make a ticket due to troubleshooter timing out, go back and click Network Status.

    No wait you get to this step which means then you made it past that part, hmm, okay I'm lost.

  • edited March 2025

    Okay so I'm going to select other buttons other than "Network Status" and hit continue, I also approved VNC screenshots, and this is on a service that's offline.

    It does look like the code update I put on here is also working as intended, it lets you immediately click submit ticket even if you do not use network status department. So it has to be some other specific issue, maybe with a specific Chicago server that's not populated properly. It'd be helpful to know what node you're on.

    @AlwaysSkint said: CHIZ001 & CHIZ035 also down

    Going to guess it's one of these, I'll check in a bit.

  • msattmsatt ProviderOG

    My 2 servers on CHIZ029 & CHIZ030 have both come back online?
    @VirMach still happy for migration or remain (Ticket #859916) whichever is more convenient for you.
    Thanks.

    Get your FREE VPS if you develop Open Source software

  • AlwaysSkintAlwaysSkint OG RETIRED
    edited March 2025

    @VirMach said: is it that the select service box looks like it already selected the service if it highlighted it?

    I get to that screenshot above..
    First few/many attempts, displays just IP. It did populate with the full service description once.
    Clicking Continue, is when the box pops up, does its status check/timeout, the only option is to close.

    Knackered typing this!

    [Firefox/Linux of course and Ghostery paused.]

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • AlwaysSkintAlwaysSkint OG RETIRED
    edited March 2025

    Feckin worked that time! To a standard looking Ticket submission page. :/
    Title:Migration

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • @AlwaysSkint said:
    Feckin worked that time! To a standard looking Ticket submission page. :/
    Title?

    That's because the servers are back online. Yes I'm confused too.

  • edited March 2025

    Strategy change, we're going to focus on making sure backups run immediately so we have fresh copies.

    (I'll still migrate people who requested afterward.)

  • AlwaysSkintAlwaysSkint OG RETIRED

    @VirMach said: (I'll still migrate people who requested afterward.)

    Defer mine until fresh backup has run - will save me a LOT of time/effort/grief.

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • edited March 2025

    @AlwaysSkint said:

    @VirMach said: (I'll still migrate people who requested afterward.)

    Defer mine until fresh backup has run - will save me a LOT of time/effort/grief.

    The service network is back up, we can just move the existing data from the server directly. No longer loaded from backups, we'll just straight up migrate it normally.

    I don't actually know about your service specifically our monitors are just online.

  • AlwaysSkintAlwaysSkint OG RETIRED

    @VirMach said: No longer loaded from backups, we'll just straight up migrate it normally.

    I'm kewl with that. Both services/VMs are back online.

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • Okay guys instead of confirming with everyone again whether or not they intend on migrating or not per ticket (after it returned online) since not everyone is going to want one or the other, I'm about to send out email communication. And I'm going to reply to all tickets letting everyone know to refer to email.

    Once you get the email, just reply back to the ticket to confirm your choice. Sorry if that's annoying, just cleaner way to handle it.

  • edited March 2025

    @VirMach said: That's because the servers are back online. Yes I'm confused too.

    Running to get backups.

    Virmach's team has been awesome through this offering to restore backups. I was dumb and have been using my largest VM at VM for a hobby project in Chicago the last month.... at Quadranet, without nightly backups. This one's on me, but support has been amazing through this.

    Time to go get a 3rd and fourth backup, because nfc what's going on.

  • msattmsatt ProviderOG

    @VirMach Would it not be easier to enable the migrate button ?

    Get your FREE VPS if you develop Open Source software

  • AlwaysSkintAlwaysSkint OG RETIRED

    @VirMach
    Unable to confirm without a reply from you first i.e. go ahead with migration to SJ.
    Ticket #658670

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • @VirMach, Migration Request from Chicago, Ticket Created #268145

  • AlwaysSkintAlwaysSkint OG RETIRED

    Migration went smooth; thanks @VirMach

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • Ok my ticket is now also in to migrate my 3x CHIZ to AMS, hoping for them to also have smooth sailing across the ocean =)

  • AlwaysSkintAlwaysSkint OG RETIRED
    edited March 2025

    LMFTFY

    @ZA_capetown said:
    Ok my ticket is now also in to deport emigrate my 3x CHI to AMS, hoping for them to also have smooth sailing across the ocean

    =) =)

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • First batch of migrations are about to happen for Chicago. A few were done yesterday, this will be the bulk. I'll likely keep doing them in 24 hour batches for the first few days as it was taking too long finding suitable servers per ticket. Also, fresh backups are mostly complete so that helps (migrations yesterday took longer since we were also simultaneously backing up.)

    @msatt said:
    @VirMach Would it not be easier to enable the migrate button ?

    Once the new system comes out, yes. (But that doesn't exist.) Old one, not so much. It'd probably cause more problems at this point.

    I had a little bit I wrote about your AI comment, maybe I'll save that for later.

  • imokimok OG Not Administrator

    Good morning Mr. Virmach and everybody here.

  • Morning @imok. Thanks hanks for the migration @VirMach - appreciate you and the team's fast response to this and the options given.

    FWIW, some good came of this - I'm now smarter and run a backup of the SQL DB, even for a hobby project, every hour and have it auto scp off to two other places.

  • @Kris said:
    Morning @imok. Thanks hanks for the migration @VirMach - appreciate you and the team's fast response to this and the options given.

    FWIW, some good came of this - I'm now smarter and run a backup of the SQL DB, even for a hobby project, every hour and have it auto scp off to two other places.

    n00bz

  • Migration from Chicago to San Jose worked very good, it's online. Thanks @VirMach

  • AlwaysSkintAlwaysSkint OG RETIRED

    @tenpera said: Migration from Chicago to San Jose worked very good..

    Shiite, I've got a stalker! :o

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

  • Well, crap. My free 2G for life is on CHIZ001 and it's been the fastest and closest of my virmach boxes. I'm near the middle of the USA, so that was a good location for me. My other two are in ATL which is nearly as close (RTT-wise), but all three in one location seems a bit risky. OKC sounded great, but I gather those plans are on hold?

    Maybe I'll just ride things on in Chicago?

    Either way, I've got backups so if it does get migrated, a full wipe/recreate is fine, especially if it saves virmach any hassle.

  • AlwaysSkintAlwaysSkint OG RETIRED

    @JBB said: CHIZ001

    IME, one of, if not the most stable VPS node in my lineup. Sad to see it go.

    In stasis until the shitposting stops/abates.
    Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
    It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).

Sign In or Register to comment.