I have seen a couple questions about ZFS and TrueNAS scaling over on the TrueNAS forums and figured I would recount a large file server I designed and implemented using ZFS back in 2008 or so.
The customer had a requirement for one large SMB share with a capacity for well over a million files and 50TiB. Remember, this is 2008 and large hard disk drives (HDD) were 1TB. The only filesystem we found that would scale that far and not have serious performance issues was ZFS. They only needed 3 or 4 users accessing it at one time.
The final design ended up being a Sun M5000 server with 4 SPARC64 CPUs and 64GiB of RAM. We really should have put more RAM in, but 64GiB seemed a good choice and aligned well with the recommendation at the time of 1GiB RAM per TB of storage.
We had 5 JBOD chassis each with 24 HDD (750 GB) all dual SAS attached via four 4-lane SAS ports on the server. JBODs 1,2,3 shared a pair of SAS ports and JBODs 4,5 shared the other pair of SAS ports. We used RAIDz2 with 5 HDD per vdev. All of the HDD in slots 0 and 1 were hot spares. All of the HDD in slot 2 were in a single vdev, same for slot 3, and slot 4, etc. up through slot 23. So we had 22 RAIDz2 vdevs all in one zpool. During testing, a technician accidentally pulled power from a JBOD. ZFS kept running, with reduced performance. As soon as power was restored, the zpool resilvered and everything was back to normal in a manner of minutes.
To summarize:
- 1 zpool
- 22 top level vdevs, all RAIDz2
- 5 HDD per vdev
- 10 hot spare HDD
- Total of 120 HDD (not counting the 2 OS boot HDD)
- Net capacity of about (0.75TB x 3 x 22) 49.5TB
Of course, whenever a HDD failed and a hot spare replaced it you lost the intended striping across all chassis until the original drive was physically replaced and then logically replaced the hot spare in the vdev.
How did we back this up? Via snapshots and a ZFS send / receive script to a smaller (higher density) and lower performance server. We also backed it up to tape (via Netbackup and some large tape libraries), but we estimated 3 weeks to perform a full restore, so designed it to never need such.
