Your advice: How to store 20 TB of data (Raid Array) at home?
Dear fellow LESbians,
my collection of digital media files grew exponentially during the Covid-19 pandemic. I have a rented server with 4x10 GB spinning disks in RAID-10 (= 20 usable TB in total) and the array is already filled with over 10 TB. I expect to hit the full 20 TB in some months time, but this will then also be the maximum storage, that I will need.
However, the server is not horribly expensive, but also not a bargain with monthly payments around EUR 90,00. I wonder whether it would make sense to have the data at home instead. I have a home internet connection with 100 Mbit upload capacity, so even hosting my data at home for access on the go could be an option.
What would be your choice for the job? A NAS? Something else? I have no idea what the current "state of the art" solution is. It is important to me to at least have the data in a RAID-1 array. No Raid or Raid-0 is not an option. I know that a Raid Array is no replacement for backups, but it would let me sleep better anyhow. A NAS would be my first idea, but I saw that they are not cheap and I wonder whether going on renting the server (with free hardware replacements) isn't the better choice in the end...
Thank you very much in advance for your ideas and suggestions!
Stay negative or at least keep it mild & kind regards
Amitz
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Comments
Just regarding streaming from home:
I fiddled forever with Plex since Vodafone doesn't give you a public IP. I am now using our FritzNAS (Fritzbox + any external hard drive) and while it doesn't have the looks from Plex, watching movies/series from remote works like a charm. Just your remote device probably will need IPv6 (either from your mobile carrier or the network/wifi you are in when watching). Only really tried watching on mobile devices, but you can also access the NAS in the browser and watch
We only got 25 Mbits, so you should be good.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
a bucket will work!
but seriously, how tolerant are you to heat/noise at home?
We accept Karma donations for the last flan. 🍮 affbrr
did you know that Amitz is warewolf ?
There are quite a few options, the best would be a NAS/Microserver as they tend to be quiet and have fairly low power consumption.
You can also reduce the noise and power consumption to an minimum by using the least amount of disks possible - here is a few options, all of which use the same base hardware a QNAP TS-451+.
The NAS alone costs £320 - this could easily be substituted with an old desktop PC assuming it has 4 SATA 3 ports and a gigabit NIC.
Disk wise if you wanted no redundancy you could go for a single Seagate Exos X18 18TB disk which is the quietest and least power hungry option to get almost 20TB in a 4 bay chassis at £341 for a total of £661
Add an additional Seagate Exos x18 18TB disk and use RAID 1 for redundancy for a total cost of: £902
You could also go for 4x Seagate IronWolf 10TB in a RAID 10 array for £254 per disk or £1336 including the NAS.
These are all expensive ways to get ~20TB with little redundancy but for a home/office environment it will likely be the best experience and use the least amount of power.
If power/noise isn't something that concerns you I would recommend getting desktop PC / Tower Server with at least 6 drive bays and purchasing 6x 4TB WD RED HDDs - this would cost £536 for the drives + the cost of the PC. You would then need to RAID them using RAID 5, not the fastest or most redundant but allows for a single drive failure and gets you 20TB. ZFS would be ideal for this likely using RAIDZ1.
Its almost always cheaper to rent as many providers purchase disks and servers in bulk and get a much better deal, it also means it is the providers responsibility to replace failed hardware which can prove expensive in the long run.
The ideal is to actually do both, keep a copy of your data locally and have an offsite backup.
I know what you are getting at, but I think that having a real server / not quiet computer at home would be outside my tolerance level. I really like it quiet...
I love you, but I wonder whether all this can also be done with a dog as I (by coincidence) have one at hand and am allergic to cat hair?
Hush! Hush!
Thank you very much for the summary concerning my options! I will do some calculation to see which of the possibilities would make most sense in my case and your advice is a great starting point!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I did what @natvps_uk suggested just last October, going for the pc option as a proper NAS can be slightly limited in hardware and I needed it for some other tasks at home anyways, running a few VMs.
Went for a Fractal Design Node 804 case which can hold up to 10-12 drives.
I had 4x 3TB drives in my old NAS so just moved them over.
It cost me around 500 euros without the drives but leaves me plenty of room for upgrades when it will become necessary
FreeVPS.org - Grab a free VPS! | SOGo Webmail - Powered by Alinto
@daffy: Excellent, thank you!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I'd start by browsing the /r/datahoarder subreddit a bit...they're constantly doing this sort of half enterprise half DIY mass data storage stuff so will be on top of the current gear recommendations
I'd also check whether you can transcode some of your linux ISOs to x265. Should make for much smaller file sizes.
If there are part of the 20TB you don't often need you could look into cold storage options. Even if you offload say 5TB it would mean less disks
I'd probably go for a TrueNAS zfs setup...handful of drives and a SDD cache.
Btw, my biggest regret is that I was a bit el cheapo with the motherboard which only has 2 ram slots and filled them with 2x8GB.
After a while now I see more and more use cases for my little machine and would have done well with 32GB.
So consider your options/budget/usecase and don't make the same mistake I did
FreeVPS.org - Grab a free VPS! | SOGo Webmail - Powered by Alinto
Great info, thank you @havoc & @daffy!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I use 4x 12TB (connected via USB3.1 in dual bays) with ZFS raidz1 and 4x2TB ssd caching it all (L2ARC).
Works really fine to back a CDN and generate thumbnails. Gives me 36TB of usable storage.
@foxone - Great, thank you! I was wondering whether simply attaching 4 drives via USB to my existing Linux PC at home would be a viable solution, too! Is there really no issue with creating a RAID 1 array with external USB drives?
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Having just done this I'd tread carefully - take a careful look at the USB power situation. Multiple nvme drives can overload the USB's aggregate ability to feed it. Worse it only shows up under load since nvme draw depends on usage. Chances are very good that it "works" but is outside of spec
TrueNAS freaked out about data issues as a result in my case
I went with an Intel build years ago, since they support C6/C7 power states. When they're in a C6/C7 power state they barely draw any power, even less than most NAS/microservers. Threw in a a ton of second hand (consumer grade) hardware (CPU, motherboard, PSU, DDR3 memory and a small SSD for booting), and shucked a couple of WD MyBooks for helium filled WD Red-like drives (some even had plain WD Reds inside). It runs Debian with a ton of services in Docker (Plex, Sonarr, Radarr, Vaultwarden, Wireguard, etc.).
I have over 27 TB of (mostly) fully reduntant storage for a little under 1000 € (a 10 TB WD MyBook gets on sale on amazon.de for 159 € every so often). One of the drives has over 7 years of power on time, and still counting. Would've been much more expensive to rent this elsewhere (although that is by no means a fair comparison, since I'm running consumer grade hardware)
What I used to do is run a HP Microserver G7 for my NAS, but now what I do is just keep the drives on a shelf and drop them into a enclosure when I need to get at the data and index it using https://vvvapp.sourceforge.net/ (Virtual Volumes View). Works quite well if you don't always need instant access to the data and are just looking to archive stuff.
Cheap dedis are my drug, and I'm too far gone to turn back.
The more I think about it... Maybe I could do it really simpler (and cheaper). If I indeed convert all my Linux ISOs to H.265, then 10TB of usable disk space would be plenty enough. If I buy two external USB drives (something like 12 TB to be on the safe side), then I could fill the first one and mirror it via rsync every night to the second one. No raid, but still a comparable result. This solution would allow to just use the computer that is under my desk anyway and IF I would really like, then installing a media server on it would not be a big thing.... Mmmmh....
But this opens up a new question and I am hoping - again - for your appreciated feedback:
I see that most external USB drives of that size (approx. 10-12 TB) come from three manufacturers, at least on Amazon.
Those are Western Digital, Seagate and Intenso.
I wonder: Which of these brands is seen as the most reliable nowadays? Or does it not matter, as they are all equally shitty in 2022? Does anyone have experience with those "WD Mybook" or "WD Elements" products? It seems as if they offer more than I need. Two basic large USB HDDs with cases would suffice.
Again, thank you all in advance!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Please do keep in mind that support for x265 playback is still limited, although it does save a lot of space.
WD Mybook and WD Elements both contain the same drives, they're whitelabeled WD Reds, and built for running 24/7. The only difference is the Mybooks come with hardware level encryption. Most drives are not built for running 24/7, especially not external drives, so I'd recommend going with WD's. I've shucked quite a few of them and they never failed on me. If you want to use the computer under your desk you might consider shucking them, as computer cases generally provide better cooling and less risk of tripping over USB cables, resulting in potential data loss.
I can agree. WD's have been really stable for me. All drives I used for NAS/Plex have been WD Elements so far. All still going strong.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
I don't have made any experiences with WD's external drives yet, but with Seagates HDDs, I'm running currently 2x 4TB Expansion(+) Drives 24/7 in my home network and they are pretty good.
this is probably the easiest approach and mirroring via some automated copy should be easy enough. also saves you the hassle with managing the raid and possible unneeded rebuilts being triggered by power loss or the like (which would try to recopy the whole data).
for the brands I'd say it does not matter so much anymore nowadays, which brand you choose, but rather check what kind of drive is inside. all brands have different drives for different use cases, like WD red green blue black and so on and these might even have different speeds (5400/5900/7200 upm), different cache size and eventually different technologies ( google: cmr / smr ).
from what you listed I think Intenso is not a vendor for disks as such but more likely a reseller who only puts its own label on it. so inside you may find different drives/brands sometimes even for the very same product. usually you can find some discussions or reviews about that from people who tend to buy external drives to dismantle and use the drive seperately (because those are often cheaper compared to buying internal drives).
You guys are wonderful and incredibly helpful, thank you very much! I will check the available free bays in my desktop computer to see whether I can fit two additional drives in there, otherwise I will go the WD external route...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
no happy ending without the Amitz dance.
There you go:
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Why so complicated? Buy a used server on eBay where you can simply put in your drives and send it in for colo? Should be around 40-50€/m.
+1 for the MyBooks. Super solid and reliable, though you can mostly achieve the same result with a WD Red drive and a USB Enclosure if you want to save a bit of money. Can't go wrong either way.
Cheap dedis are my drug, and I'm too far gone to turn back.
This is a very common use-case over at ServerBuilds, and a number of whitebox NAS builds would be suitable. E.g., $40 LGA1155 board in a large case like RSV-L4500 (15 bays) using SAS2 HBA and either cheap SAS spinners ($10/TB) or WD shucks (MyBook/Elements), running Unraid (or snapraid+mergerfs if you like). The WD shucks are HGST datacentre drives and very solid. Unraid makes it very easy to incrementally add drives of mixed capacity.
Plex is offloaded to a $100 SFF desktop with a 9th-gen Celeron, using QSV hardware transcoding of 1080p source for remote users. These little corporate desktops can do 21 simultaneous 1080p transcodes while drawing only 10W. 4k content (remux) is in a separate Plex instance on the NAS with transcoding disabled (direct-stream takes almost no resources on the NAS).
Many of our users have over 100TB at home using these builds. Pricing above is for the US, of course; a number of our EU users opt for newer NAS builds with QSV. Feel free to join us on Discord; I'm a mod there.
RAID/Unraid is not backup; maintain automated 3-2-1, etc., as you well know.
I don't know if you are from the UK or not, but I'm with Vodafone for VDSL and have the 80mbps pro plan for £23 a month, they will give you a free static IP if you request one (I have one).
Thanks for bringing that up! Unfortunately, I am from Germany
When I last called them and asked, they said that they won't do this anymore.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
I personally never use RAID at home. I like to keep things dead simple.
But then I barely have 5TB of data to keep. Not much of a sentimental person, I am.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.
You won't catch bit rot with this though. Might be an acceptable risk in this case though
not true. at least if you are on cable. don't ask for a static IPv4 but for 'switching from DS lite to full dual stack' ;-)
afaik it incur some small fees, unless you are in the middle of a bigger complaint, then they're quite happy to give to you for free... or so I heard
But dual stack doesn't mean static IP rather dedicated IP so basically what you've got with an old DSL16.000 service.
that's true. but I have yet to see the IP change, even on a reconnect it stays. and with cable there is forced reconnect after a day anymore, so essentially you stay online anyway and the IP doesn't change. static enough for me... ;-)
No. Just remember to never use /dev/sdX devices, and use /dev/disk/by-id/xxx so that even if they get detected at different times they don't switch names.
Also, with
lsusb -tyou can see the internal hub tree of your server. Try to connect the drives at 10G (or dedicated 5G) hubs, so they don't get bottlenecked. Basically, never connect more devices than the speed your port can handle.In my case:
All drives are powered externally with a beefy industrial 12v /5v power supply.
That's ideal. I went with USB powered and yeah not the most inspired of plans..4th one added started to cause issues
Sooo... I went ahead and bought one WD Elements 14 TB unit to do some testing and to start downloading all stuff from the server. I do not even want to think about how long this will take (I am innerly prepared for days if not weeks). When I am finished and happy with the result and performance, then I will buy a second one to do the backup.
Does anyone have an idea how long 10 TB might take with a 500 Mbps connection that is nearly maxed out? Are there calculators for something like this out there? I must admit being to lazy to calculate it myself, even though I have the general tool (brain) and apps (math knowledge) at hand... ;-)
//Edit:
Scratch that: I found a calculator. It says around 44 hours. Now that would be absolutely bearable...
Stupid question, but I am out of the Windows world since WindowsXP - The drive contained a folder named "System Volume Information" and was pre-formatted in NTFS. NTFS is fine for me, but I assume that I can safely delete this folder, right? I am using the drive exclusively under Linux and maybe MacOS. That "System Volume Information" folder is a Windows thing for snapshots & stuff, isn't it?
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I have also shucked Western Digital drives. Cost today is about $17 per TB, and was as low as $14.50/TB during the November/December shopping time. Voids the warranty, but cheaper than internal drives. Used five shucked 8 TBs for two years, and recently sold them after upgrading to three 14 TBs.
☝ This. Glad I went from 5 to 3 online hard drives. Three drives are as much noise I could tolerate in my home office.
You can also use a hot-swappable bay for plugging in/out drives when needed. For things like old downloads that are replaceable by downloading again, no need to be RAIDed and powered on all the time. I actually don't use RAID at all, just cron jobs that sync important directories across all 3 drives, which saves the disk space needed to mirror everything and saves the hassle of long-term RAID maintenance. Also, about 2x/year I manually backup stuff to a hot-swap drive that I keep elsewhere in the house, in case fire or theft in the office.
level of laziness: premium
it's 10 * 1024 * 1024 * 8 / 500 / 60 / 60 = 46.6 hours - consider overhead and shit and add 20% or so but still, 2-3 days only ;-)
Maybe it's what you found, but this is the one I use: http://www.dslreports.com/calculator
mind the bits (bps) vs bytes (Bps) per second in the speed box
Format the drive, use in Linux, Mac, whatever. Don't leave as NTFS unless using on Windows.
This will depend on how important your data is. a Simple NAS would be enough especially if you don't need a noisy server at home
Web Hosting - VPS Cloud - EU Dedicated Offers Managed Hosting Made Easy Dedicated Servers, Private Cloud, Colocation, VPS, Shared & Reseller Hosting
Well, I will let it as NTFS. I can read & write it fine under Linux and macOS and I could still take it with me to Windows friends (the few I have, strange guys) and they can read the disk without issues. Same with my Samsung TV. I am not sure whether it accepts something at EXT4 or APFS. It might, but I never tried.
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
/me suffers in GNU
I have 2 * 14TB WD elements and 2 * 8TB in my Synology ds920+ using Synology's hybrid raid which gives me about 25TB of usable space.
Been running these drives for sometime including using them in a custom built home server that used mergerfs and snapraid, not experienced any issues with them.
If I have learned something during my years using linux, then that there is always someone suffering in GNU.
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
GNU is Not Unsuffering
blog archives
The easy solution is to get a NAS with enough (is it ever enough?) disk space and use that.
Silent and accessible from almost everywhere.
“Technology is best when it brings people together.” – Matt Mullenweg
18TB WD Elements is $280 today (in the US). Search WDBWLG0180HBK-NESN on WesternDigital.com, price changes after added to the cart. Assuming this one's shuckability is the same as the smaller guys, $15.56 per TB is a good price.
Checked my prices today for a
Total Price, converted to USD = $3,256
But that would be enough for some time ....... 54TB with Synology Raid (which is software Raid5)
“Technology is best when it brings people together.” – Matt Mullenweg
Seagate is now shipping 22TB SMR HDDs.
Source.