Ok, official info was sent to remaining affected customers, please check your e-mails.
Here is a time-lined description of the event of the fuckup:
July 27, 2025 – Afternoon (GMT+3):
Multiple Seagate ST18000NM019J drives (firmware KM02) across two nodes suddenly powered down due to a firmware-related failure. Drives began reporting critical SMART alerts (Data channel impending failure), causing the RAID-6/60 array to become unavailable.
Result:
Addon storage volumes became inaccessible, and VPS services depending on those volumes were disrupted. Some NVMe-based systems also experienced write issues due to OS-level I/O buffering.
July 28, 2025 – Morning:
Our team accessed the datacenter, identified the fault, and began recovery efforts. All NVMe-only VPS services were successfully migrated to healthy nodes.
July 28–29, 2025:
RAID array access was restored in degraded mode, enabling partial access to addon volumes at limited transfer speeds.
RAID controller entered fault mode due to concurrent SMART failures
No physical disk damage, no reallocated sectors or ECC errors — this was purely firmware-triggered
🛡️ Mitigation Going Forward
We are conducting a full infrastructure audit to identify any remaining ST18000NM019J drives with KM02 firmware
Affected drives will be proactively replaced or updated, where supported
RAID monitoring thresholds and firmware validation processes are being tightened to catch these failures earlier
This was an unprecedented firmware-level failure that bypassed typical RAID fault tolerance. We appreciate your understanding as we finalize recovery efforts for impacted systems.
Here is an output of one of the drives, maybe it can help others to check theirs if they have the same model used, all 6 reported exactly the same error, have the same powered on hours ( ~266 days ) and were brand new.
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST18000NM019J
Revision: KM02
Compliance: SPC-5
User Capacity: 18,000,207,937,536 bytes [18.0 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500d8a51a07
Serial number: ZR57B8800000G20806CV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Mon Jul 28 17:36:48 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: Data channel impending failure general hard drive failure [asc=5d, ascq=30]
Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 31 C
Drive Trip Temperature: 60 C
Accumulated power on time, hours:minutes 6367:42
Manufactured in week 01 of year 2022
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 34
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 291
Elements in grown defect list: 1
Vendor (Seagate Cache) information
Blocks sent to initiator = 3828
Blocks received from initiator = 1650689
Blocks read from cache and sent to initiator = 9094
Number of read and write commands whose size <= segment size = 29
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 6367.70
number of minutes until next internal SMART test = 53
Seagate FARM log supported [try: -l farm]
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 0.016 0
write: 0 0 0 0 0 6.889 0
Non-medium error count: 0
Pending defect count:0 Pending Defects
The error in bold triggered the detach of the drives from the raid array.
Here is a screen shot from the log of one of the dells servers ( R740 ) showing 2 drives leaving the " chat" at the precise same time, DST was not set on the server so that is why the time shows only 12:00
Upgraded my 5TB VPS from Debian 12 to 13 and the networking died. Make sure you know your root password in advance, the control panel can't change it, at least on debian 13, and you need it for VNC. Thankfully I found it lying around.
To fix networking, put this in /etc/netplan/99-static.yaml (and chmod 600):
I think doing this after upgrade but before the post-upgrade reboot should prevent death of networking. Get addresses/gateway from control panel. Don't set v6 gateway, it'll break IPv6. After upgrade my server was set to use DHCP for some reason, which didn't get any addresses.
I don't actually know why it broke but this is working across reboots. nameservers might be unnecessary but when networking first died it also wiped DNS server setup, it didn't have any set at all.
If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
It’s OK if you disagree with me. I can’t force you to be right!
Looking at the CPU, we are safe for now.
We don't have a P**N Collection — we search YouTube for "buff dudes" or "pamela reif" whenever we feel like it.
Our name and email and telephone are all real, but address is office mailing address — residential address can be provided in exchange for a hoodie.
@yoursunny said:
Looking at the CPU, we are safe for now.
We don't have a P**N Collection — we search YouTube for "buff dudes" or "pamela reif" whenever we feel like it.
Our name and email and telephone are all real, but address is office mailing address — residential address can be provided in exchange for a hoodie.
Interesting fact, we actually had close to 0 problems like the ones described in the mail with the vast majority of the "active" users from here and OGF.
@host_c said: Interesting fact, we actually had close to 0 problems like the ones described in the mail with the vast majority of the "active" users from here and OGF.
Is there a reason you wouldn't talk about it here? I haven't seen the email so I don't understand what problem we're referring to.
Well, I think it is time we do not use forums for support tiket , tho this falls under announcements, here it is:
A Look Back at 2025 and the Road Ahead
Dear customers,
As many of you know, we entered 2025 with an ambitious list of improvements we wanted to deliver. Unfortunately, many of these plans did not materialize. Over the past few days, I took the time to analyze why.
The conclusion was straightforward: too much of our time was consumed by daily abuse cases, fake accounts, and operational distractions, leaving little room for the larger and more complicated projects we committed to.
And let’s be honest — 2025’s geopolitical and economic chaos did not help either. 🙂
For 2026, this will change.
Zero tolerance for fake accounts
We have repeatedly asked customers to update their profiles with real information. Some complied, many ignored the request.
To protect our platform and eliminate wasted time, we will enforce a 0-tolerance policy for fake accounts starting immediately.
Additionally:
→ We will tighten our fraud-protection rules.
After more than 12 months of operating our automated fraud-detection system, we can confirm its accuracy:
9 out of 10 flagged orders were indeed abusive
Orders manually approved just 1% above the fraud threshold turned abusive 9 out of 10 times
Therefore:
→ Borderline and suspicious orders will no longer be manually approved.
→ Fraud screening will be stricter and strictly enforced.
The TOS will be updated shortly.
Product naming issues & the $6 upgrade situation
We acknowledge that some product naming and upgrade communications in 2024–2025 caused confusion — some of it chaotic:
The Pioneer 5TB LET/LES editions created ambiguity (now corrected).
The upgrade from E5-V4 → Scale Gen1 is still misunderstood by some customers as an upgrade to Gen2.
In practice:
Scale Gen1 severely underperformed
It was retired quickly
Impacted customers were temporarily moved to Scale Gen2
This created a fairness issue:
Some customers paid $6 for an upgrade that did not provide long-term value, ending up on Scale Gen2, while others paid higher prices for native E5-V4 → Scale Gen2 plans.
To restore fairness and consistency:
→ All affected customers who paid the upgrade from E5-V4 to Scale Gen1 will receive a full $6 refund as account credit.
→ Services will be migrated back to their initial E5-V4 platform until final decommissioning.
→ This is mostly the case for the Pioneer 5TB and Nimitz 10TB plans.
This takes effect immediately as of 04-12-2025.
This corrects the imbalance, and no one loses anything. This was one of the largest operational mistakes of 2024–2025, and we have learned from it.
Abuse of promotional products & transfer restrictions
A large portion of platform abuse occurred on promotional products — often transferred repeatedly between users.
Some services changed owners five or more times in just a few months, and nearly every time ended up abused.
As it turns out, our initial wish to be flexible in this area came back to bite us.
Sometimes it is better to be more rigid, and flexibility should not be present in all parts of the service.
This is also why we did not participate in this year’s LET and LES Black Friday sales.
Until we develop a reliable anti-abuse formula, participating is a waste of resources and creates discomfort for existing customers.
Starting today, 04-12-2025:
→ Promotional products will no longer be transferable.
For list-priced (non-promo) services:
→ Transfers remain allowed but must be requested via support ticket.
There will be no other exceptions.
The TOS will be updated accordingly.
End-of-life for E5-V4 services
As previously announced, all E5-V4 services will be decommissioned by the end of 2025, with two exceptions:
Pioneer 5TB
Nimitz 10TB
There will be no other exceptions.
The E5-V4 platform is outdated, unstable, difficult to debug, and performs poorly on the 2025 hypervisor stack. It is no longer viable to maintain.
Looking ahead to 2026
2026 is just around the corner and, while I could say much more, I will keep it simple:
→ More attention to detail
→ Improved stability
→ Increased performance
We will roll out additional storage options using Hardware RAID10, correct mistakes of past years, bring new features, and focus on quality over all other aspects.
For those who stay with us: thank you, and a big hug.
Your support and patience mean a lot to us.
Happy holidays, and best wishes from our entire team.
Really appreciate the update! I have a quick clarification question about the 5TB storage plans -- if my plan is labeled now as "Pioneer XNova - Storage 5TB - I185 - G2" does that mean I'm on Gen 2, and should be good to go to keep using it indefinitely? Or are all the Pioneer plans regardless of generation going to eventually go away, I swear I remember reading something about 2027 in the email?
(edit: I checked the email, says nothing about that)
@SocksAreComfortable said: if my plan is labeled now as "Pioneer XNova - Storage 5TB - I185 - G2" does that mean I'm on Gen 2, and should be good to go to keep using it indefinitely?
yappp.
well, I will not say indefinitely, as in 10 years from now i highly doubt we will run Scale Gen 2. , but the G2 will not be scheduled for decommission anytime soon.
Once I will redo all the TOS from ground UP, it will be sent out to all active customers.
Comments
Ok, official info was sent to remaining affected customers, please check your e-mails.
Here is a time-lined description of the event of the fuckup:
July 27, 2025 – Afternoon (GMT+3):
Multiple Seagate ST18000NM019J drives (firmware KM02) across two nodes suddenly powered down due to a firmware-related failure. Drives began reporting critical SMART alerts (Data channel impending failure), causing the RAID-6/60 array to become unavailable.
Result:
Addon storage volumes became inaccessible, and VPS services depending on those volumes were disrupted. Some NVMe-based systems also experienced write issues due to OS-level I/O buffering.
July 28, 2025 – Morning:
Our team accessed the datacenter, identified the fault, and began recovery efforts. All NVMe-only VPS services were successfully migrated to healthy nodes.
July 28–29, 2025:
RAID array access was restored in degraded mode, enabling partial access to addon volumes at limited transfer speeds.
🧪 Root Cause
Firmware fault affecting multiple ST18000NM019J (KM02) drives simultaneously
RAID controller entered fault mode due to concurrent SMART failures
No physical disk damage, no reallocated sectors or ECC errors — this was purely firmware-triggered
🛡️ Mitigation Going Forward
We are conducting a full infrastructure audit to identify any remaining ST18000NM019J drives with KM02 firmware
Affected drives will be proactively replaced or updated, where supported
RAID monitoring thresholds and firmware validation processes are being tightened to catch these failures earlier
This was an unprecedented firmware-level failure that bypassed typical RAID fault tolerance. We appreciate your understanding as we finalize recovery efforts for impacted systems.
Here is an output of one of the drives, maybe it can help others to check theirs if they have the same model used, all 6 reported exactly the same error, have the same powered on hours ( ~266 days ) and were brand new.
SMART Health Status: Data channel impending failure general hard drive failure [asc=5d, ascq=30]
The error in bold triggered the detach of the drives from the raid array.
Here is a screen shot from the log of one of the dells servers ( R740 ) showing 2 drives leaving the " chat" at the precise same time, DST was not set on the server so that is why the time shows only 12:00
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”
Hope all is well with @host_c (Last Active: September 17). Just renewed my awesome 5TB VPS for another quarter
LinuxFreek.com — Hosted on 🇪🇺 Scaleway Stardust with Native IPv6 | IPv4 Proxy, WAF & DNS powered by 🇳🇱 DutchIS
Surely those euro summer vacations are over by now?!? 😬🤷♂️
Rock Solid Web Hosting, VPS & VDS with a Refreshing Approach - Xeon + EPYC with DDoS protection and Enterprise Hardware! HostBilby Inc.
I hope so
2CPU, 2GB RAM, 35 GB SSD = $17.66/y (aff) | Direct Admin Shared & Reseller (aff) | 1, 2, 4, 8, 16 TB StorageBox (aff) | 2CPU, 4GB RAM, 65GB NVMe = €24/y (aff)
They are busy upgrading and fixing things. The team is working on it and I am sure they will be ready for the next few months soon.
Upgraded my 5TB VPS from Debian 12 to 13 and the networking died. Make sure you know your root password in advance, the control panel can't change it, at least on debian 13, and you need it for VNC. Thankfully I found it lying around.
To fix networking, put this in
/etc/netplan/99-static.yaml(and chmod 600):I think doing this after upgrade but before the post-upgrade reboot should prevent death of networking. Get addresses/gateway from control panel. Don't set v6 gateway, it'll break IPv6. After upgrade my server was set to use DHCP for some reason, which didn't get any addresses.
I don't actually know why it broke but this is working across reboots.
nameserversmight be unnecessary but when networking first died it also wiped DNS server setup, it didn't have any set at all.I reserve the right to license all of my content under: CC BY-NC-ND. Whatever happens on this forum should stay on this forum.
This is my fav backup server, ever!
Amadex • Hosting Forums
@Amadex how much per year?
2CPU, 2GB RAM, 35 GB SSD = $17.66/y (aff) | Direct Admin Shared & Reseller (aff) | 1, 2, 4, 8, 16 TB StorageBox (aff) | 2CPU, 4GB RAM, 65GB NVMe = €24/y (aff)
0€, @host_c likes me
Amadex • Hosting Forums
I'm jealous
2CPU, 2GB RAM, 35 GB SSD = $17.66/y (aff) | Direct Admin Shared & Reseller (aff) | 1, 2, 4, 8, 16 TB StorageBox (aff) | 2CPU, 4GB RAM, 65GB NVMe = €24/y (aff)
NO reason to be jealous man . His penis is still smaller than yours.
If you want information, feign ignorance reply with the wrong answer. Internet people will correct you ASAP!
It’s OK if you disagree with me. I can’t force you to be right!
I used to write songs for him
Amadex • Hosting Forums
Cause I knows everything man and I do mean everything. I am Yeti Claws after all
Oh noes, tomorrow 27.11 there's a maitenance. Machines pwoered off. Billions lost. Deadpool is at the corner.
Yeape Stu is getting drunk and they know what happens then.
Kudos @host_c for the long, insightful email
. All the best for 2026. 
Right? Sounds like they are making good decisions and we keep informed.
THX, I appreciate it.
Those that got the mail now know why we were kinda off this year and absent in the Deal Threads.
And Yes, I did use AI to correct my typing, you should have seen the draft, not even I understood it
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”
Most will actually benefit from these decisions in the mid-long run, some will not. It is what it is.
Just be sure to have some Pop-Corn on idle, I do expect some fallout in Late December / beginning of January titled:
Shit HOT-C deleted my VPS and I lost my P**N Collection - The tragedy
Witch is espected as he/she/it's fake e-mail did not forward the info.
Either-way, I am looking forward for next year.
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”
Looking at the CPU, we are safe for now.
We don't have a P**N Collection — we search YouTube for "buff dudes" or "pamela reif" whenever we feel like it.
Our name and email and telephone are all real, but address is office mailing address — residential address can be provided in exchange for a hoodie.
vps9hostname is available. affbrrInteresting fact, we actually had close to 0 problems like the ones described in the mail with the vast majority of the "active" users from here and OGF.
And I am proud of that!
PS:
Anyone should have one sincerely.
Cheers!
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”
Asciip0rn rocks !!
https://github.com/gabe565/ascii-movie
blog | exploring visually |
that is pretty cool actually.
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”
Is there a reason you wouldn't talk about it here? I haven't seen the email so I don't understand what problem we're referring to.
Well, I think it is time we do not use forums for support tiket , tho this falls under announcements, here it is:
A Look Back at 2025 and the Road Ahead
Dear customers,
As many of you know, we entered 2025 with an ambitious list of improvements we wanted to deliver. Unfortunately, many of these plans did not materialize. Over the past few days, I took the time to analyze why.
The conclusion was straightforward: too much of our time was consumed by daily abuse cases, fake accounts, and operational distractions, leaving little room for the larger and more complicated projects we committed to.
And let’s be honest — 2025’s geopolitical and economic chaos did not help either. 🙂
For 2026, this will change.
Zero tolerance for fake accounts
We have repeatedly asked customers to update their profiles with real information. Some complied, many ignored the request.
To protect our platform and eliminate wasted time, we will enforce a 0-tolerance policy for fake accounts starting immediately.
Additionally:
→ We will tighten our fraud-protection rules.
After more than 12 months of operating our automated fraud-detection system, we can confirm its accuracy:
9 out of 10 flagged orders were indeed abusive
Orders manually approved just 1% above the fraud threshold turned abusive 9 out of 10 times
Therefore:
→ Borderline and suspicious orders will no longer be manually approved.
→ Fraud screening will be stricter and strictly enforced.
The TOS will be updated shortly.
Product naming issues & the $6 upgrade situation
We acknowledge that some product naming and upgrade communications in 2024–2025 caused confusion — some of it chaotic:
The Pioneer 5TB LET/LES editions created ambiguity (now corrected).
The upgrade from E5-V4 → Scale Gen1 is still misunderstood by some customers as an upgrade to Gen2.
In practice:
Scale Gen1 severely underperformed
It was retired quickly
Impacted customers were temporarily moved to Scale Gen2
This created a fairness issue:
Some customers paid $6 for an upgrade that did not provide long-term value, ending up on Scale Gen2, while others paid higher prices for native E5-V4 → Scale Gen2 plans.
To restore fairness and consistency:
→ All affected customers who paid the upgrade from E5-V4 to Scale Gen1 will receive a full $6 refund as account credit.
→ Services will be migrated back to their initial E5-V4 platform until final decommissioning.
→ This is mostly the case for the Pioneer 5TB and Nimitz 10TB plans.
This takes effect immediately as of 04-12-2025.
This corrects the imbalance, and no one loses anything.
This was one of the largest operational mistakes of 2024–2025, and we have learned from it.
Abuse of promotional products & transfer restrictions
A large portion of platform abuse occurred on promotional products — often transferred repeatedly between users.
Some services changed owners five or more times in just a few months, and nearly every time ended up abused.
As it turns out, our initial wish to be flexible in this area came back to bite us.
Sometimes it is better to be more rigid, and flexibility should not be present in all parts of the service.
This is also why we did not participate in this year’s LET and LES Black Friday sales.
Until we develop a reliable anti-abuse formula, participating is a waste of resources and creates discomfort for existing customers.
Starting today, 04-12-2025:
→ Promotional products will no longer be transferable.
For list-priced (non-promo) services:
→ Transfers remain allowed but must be requested via support ticket.
There will be no other exceptions.
The TOS will be updated accordingly.
End-of-life for E5-V4 services
As previously announced, all E5-V4 services will be decommissioned by the end of 2025, with two exceptions:
There will be no other exceptions.
The E5-V4 platform is outdated, unstable, difficult to debug, and performs poorly on the 2025 hypervisor stack. It is no longer viable to maintain.
Looking ahead to 2026
2026 is just around the corner and, while I could say much more, I will keep it simple:
→ More attention to detail
→ Improved stability
→ Increased performance
We will roll out additional storage options using Hardware RAID10, correct mistakes of past years, bring new features, and focus on quality over all other aspects.
For those who stay with us: thank you, and a big hug.
Your support and patience mean a lot to us.
Happy holidays, and best wishes from our entire team.
HOST-C
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”
@host_c
Really appreciate the update! I have a quick clarification question about the 5TB storage plans -- if my plan is labeled now as "Pioneer XNova - Storage 5TB - I185 - G2" does that mean I'm on Gen 2, and should be good to go to keep using it indefinitely? Or are all the Pioneer plans regardless of generation going to eventually go away, I swear I remember reading something about 2027 in the email?
(edit: I checked the email, says nothing about that)
yappp.
well, I will not say indefinitely, as in 10 years from now i highly doubt we will run Scale Gen 2.
, but the G2 will not be scheduled for decommission anytime soon.
Once I will redo all the TOS from ground UP, it will be sent out to all active customers.
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
“If it can’t guarantee behavior under load, it doesn’t belong in production.”