The first 5TB are transferred and it seems as if Vodafone (my cable internet provider) or Hetzner is slowly starting to throttle my connection to half of its speed. It suddenly dropped from an average of 495 Mbps over days to constant 250 Mbps. Strange, strange. Well, I've got time...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Thank you so much for your empathy! I will try that!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
@Falzo: Thank you, that worked indeed! Back to 495 Mbps! Let's see how long the fun lasts...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
@Amitz said: @Falzo: Thank you, that worked indeed! Back to 495 Mbps! Let's see how long the fun lasts...
yeah I am not sure which part of the chain is the actual weakest. I can see the DOCSIS 3.1 channel for upload dropping randomly and if that starts to happen it usually goes downhill quickly, bad connection/packet loss and so on. somehow the renegotiation seems to fix it, or maybe it is some kind of overflowing buffer somewhere or whatever. glad it worked for you!
@Falzo: Thanks for the explanation! I was already wondering why this could have happened, as there is no "official" cap on transfer volume (not even a "fair usage" clause) on this business contract. I assumed that it might be some kind of automatism after a certain bandwidth has been used, but obviously it not the case.
I have monitored both upload and download rates during the process and: The download rate dropped (and stayed there) right after the upload capacity was saturated (50 Mbps) for around 15mins and did not return to normal afterwards. So your theory seems very plausible.
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I don't know if cable has the same problem as VDSL with Bufferbloat, but if I don't cap my upload/downloads on pfSense and let it upload/download at max speed the level of CRC error on my line triggers a reset on the ISP side and it lowers the line speed to reduce the error rate.
I had to setup traffic shaper (FQ_CODEL) to 90% of my actual line capacity to avoid resets.
In the US at least, shucking does not void warranty; folks have shipped the bare drive back to WD and gotten warranty replacements. My previous post included a link to a price-tracking site for shucks; I believe 12TB or 14TB is the sweet spot for $/TB nowadays.
FWIW, I have 4x12TB and 1x14TB using snapraid and drivepool.; windows reports 43.6TB usable. All are still in the enclosures and plugged in via USB3, using external power supplies to my Windows 11 PC which is plenty beefy enough to handle plex transcodes if needed. I don't share out plex, just me/family and 95% is direct play LAN traffic.
my collection of digital media files grew exponentially during the Covid-19 pandemic. I have a rented server with 4x10 GB spinning disks in RAID-10 (= 20 usable TB in total) and the array is already filled with over 10 TB. I expect to hit the full 20 TB in some months time, but this will then also be the maximum storage, that I will need.
There are other considerations though, in my view. Resiliency, EMP-proof DC facilities, electricity, Exchange redundancy, etc. These things do justify the investment in hosting vs SPoF at home.
@bdl said:
but seriously, how tolerant are you to heat/noise at home?
I know what you are getting at, but I think that having a real server / not quiet computer at home would be outside my tolerance level. I really like it quiet...
I only have it connected intermittently for when I'm actually doing backups, but a raspberry pi plus powered external USB drive is a good solution for this, especially if you actually are only going to be plugging one or two drives in at once.
I wouldn't recommend a bus-powered external USB drive though.
Old thread but it's hard to beat Hetzner Storage box for hassle avoidance at affordable cost (about €2 per TiB/month = $2 per 1e12 bytes/month since that's how HDD capacity is stated). HDD's at home have IME been horribly unreliable, especially when not kept spinning (i.e. keep drive in drawer for a year = failure on attempt to power up).
It's unfortunate that LTO8 tape drives are so expensive since the tapes (12TB capacity) are about $60 each now, and are better than HDD's for archiving. In fact they are a reasonable way to store a large media library if you can live with ~ 1 minute retrieval time for less frequently used items. You'd cache the more frequently used ones on HDD.
Anyway, Reddit has r/datahoarder for this general topic.
@willie said:
HDD's at home have IME been horribly unreliable, especially when not kept spinning (i.e. keep drive in drawer for a year = failure on attempt to power up).
Glad it's not just me. Nearly every drive I've had fail catastrophically has been one that was retired in full working order for an upgrade, then brought back into service for some other project after many months without power. Has made me very wary of using HDDs for any kind of [literally] cold storage.
Comments
The first 5TB are transferred and it seems as if Vodafone (my cable internet provider) or Hetzner is slowly starting to throttle my connection to half of its speed. It suddenly dropped from an average of 495 Mbps over days to constant 250 Mbps. Strange, strange. Well, I've got time...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
feel you bro... try restarting the router in between so the cable connection fully resyncs... seems to help some times for me.
Thank you so much for your empathy!
I will try that!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Imagine resilvering those
@Falzo: Thank you, that worked indeed!
Back to 495 Mbps! Let's see how long the fun lasts...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
yeah I am not sure which part of the chain is the actual weakest. I can see the DOCSIS 3.1 channel for upload dropping randomly and if that starts to happen it usually goes downhill quickly, bad connection/packet loss and so on. somehow the renegotiation seems to fix it, or maybe it is some kind of overflowing buffer somewhere or whatever. glad it worked for you!
@Falzo: Thanks for the explanation! I was already wondering why this could have happened, as there is no "official" cap on transfer volume (not even a "fair usage" clause) on this business contract. I assumed that it might be some kind of automatism after a certain bandwidth has been used, but obviously it not the case.
I have monitored both upload and download rates during the process and: The download rate dropped (and stayed there) right after the upload capacity was saturated (50 Mbps) for around 15mins and did not return to normal afterwards. So your theory seems very plausible.
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I don't know if cable has the same problem as VDSL with Bufferbloat, but if I don't cap my upload/downloads on pfSense and let it upload/download at max speed the level of CRC error on my line triggers a reset on the ISP side and it lowers the line speed to reduce the error rate.
I had to setup traffic shaper (FQ_CODEL) to 90% of my actual line capacity to avoid resets.
In the US at least, shucking does not void warranty; folks have shipped the bare drive back to WD and gotten warranty replacements. My previous post included a link to a price-tracking site for shucks; I believe 12TB or 14TB is the sweet spot for $/TB nowadays.
FWIW, I have 4x12TB and 1x14TB using snapraid and drivepool.; windows reports 43.6TB usable. All are still in the enclosures and plugged in via USB3, using external power supplies to my Windows 11 PC which is plenty beefy enough to handle plex transcodes if needed. I don't share out plex, just me/family and 95% is direct play LAN traffic.
There are other considerations though, in my view. Resiliency, EMP-proof DC facilities, electricity, Exchange redundancy, etc. These things do justify the investment in hosting vs SPoF at home.
Terrahost.com
Resilient Hosting -Empowering Our Clients Digitally
I only have it connected intermittently for when I'm actually doing backups, but a raspberry pi plus powered external USB drive is a good solution for this, especially if you actually are only going to be plugging one or two drives in at once.
I wouldn't recommend a bus-powered external USB drive though.
Old thread but it's hard to beat Hetzner Storage box for hassle avoidance at affordable cost (about €2 per TiB/month = $2 per 1e12 bytes/month since that's how HDD capacity is stated). HDD's at home have IME been horribly unreliable, especially when not kept spinning (i.e. keep drive in drawer for a year = failure on attempt to power up).
It's unfortunate that LTO8 tape drives are so expensive since the tapes (12TB capacity) are about $60 each now, and are better than HDD's for archiving. In fact they are a reasonable way to store a large media library if you can live with ~ 1 minute retrieval time for less frequently used items. You'd cache the more frequently used ones on HDD.
Anyway, Reddit has r/datahoarder for this general topic.
If you have tape drives at home, it's a MUST to make your basement look like this:
Glad it's not just me. Nearly every drive I've had fail catastrophically has been one that was retired in full working order for an upgrade, then brought back into service for some other project after many months without power. Has made me very wary of using HDDs for any kind of [literally] cold storage.