[Tfug] Advice on building a new machine

Dean Jones dean.jones at gmail.com
Mon Feb 20 09:15:03 MST 2012


On Sun, Feb 19, 2012 at 10:32 PM, John Gruenenfelder
<jetpackjohn at gmail.com> wrote:
> Hello again,

snippage...

> * Solid state storage
> As we need a decent amount of storage on this machine, I will be
> installing between two and four regular hard drives in either a RAID1
> or RAID5 configuration.  But, for the OS (and perhaps /home) I was
> considering a SSD.  But, I do not own one nor have I ever used one.  I
> know they're much faster, especially for random I/O, but I've always
> been a bit worried about their longevity.  Considering that the drive
> would be used for at most five years, is this something I need to be
> concerned with?  The machine will be on 24/7, of course.  Would I have
> to do anything special with regard to frequently written areas like
> /var/log?  Maybe put them on the HDD instead?  It's a solid state
> drive, but I suppose I should still put in two in a RAID1 setup, just
> to be safe?

Do not put swap on the SSD, and yes probably not /var and /var/log.

Also buy a quality ($$ probably Intel) one if you want it to last.
The cheaper ones will die within a year or two at constant usage
rates.  5 years might be a stretch, but I haven't used any for that
long and you haven't really described the kind of I/O usage on this
box.

Do some reading on TRIM and what file systems can use it best etc.  It
is a rabbit hole that can go on for days.  But the good SSDs should
handle the wear levelling.

You can also RAID 1 these devices in case of a concern for failure.

> And, while I'm at it, a couple of software/configuration questions:
>
> * File system
> I've been using XFS for just about everything since SGI first ported
> it to Linux and I had to patch the kernel myself.  It's always worked
> remarkably well, always been fast, and I've never had any data loss
> using it.  I really don't think btrfs is ready for prime time yet, but
> I'm wondering about ZFS.  As storage becomes ever more enormous, I'm
> somewhat worried about "silent" data corruption on the drives.  ZFS
> has checksums all over the place and can do periodic scrubbing to help
> alleviate this concern.  I'm looking at a total usable storage
> capacity (after RAID use is accounted for) of between two and four TB.
>  Is this large enough that I need to worry about this sort of thing?
> Unfortunately, ZFS is not in the mainline kernel and I'd hate to go
> back to rolling my own kernels again (just one more task I could do
> without).  And, the md subsystem (devmapper subsystem too, perhaps?)
> can perform a periodic scrubbing/data integrity check on its own, so
> maybe I don't need ZFS for that?  Ideally, I'd prefer to just stick
> with something I've used for a long time and that I know works very
> well, but I also don't want to lose data.  But is the risk high enough
> that I realistically need to worry about it?  And, while our
> (primarily astronomical) data is certainly important to us, we're not
> talking a life or death situation here.

ZFS on linux is a no go.  It is FUSE only last I looked  and will
likely remain that way.  Userland isn't good enough for a 24/7 file
system in my opinion.   And I LOVE ZFS and would love for it to be on
linux.

If you want ZFS you are talking about either FreeBSD or some flavor of
open Solaris.  FreeBSD lags behind in ZFS features from the Solaris
variants, but it is very stable (vs running the newer Solaris
versions).  BTRFS on linux will eventually replicate most of the
features of ZFS into the main kernel but it is still very buggy and I
advise to avoid it for now based on the bug and crash reports on the
mailing lists.


> * md (RAID/multiple disk) versus dm (devmapper) subsystems
> For a while I have been using LVM on top of an md-built RAID array as
> my usual setup for a number of systems, sometimes also throwing in a
> crypto layer.  This ties together a number of kernel systems.  It
> works, but perhaps there is a better way.  All of these subsystems are
> actively maintained, but I've read that the md subsystem can exhibit
> throughput bottlenecks.  I haven't seen any hard data, or read a
> particularly convincing explanation of this.  Since the dm subsystem
> can also handle RAID, is this a better way to organize things?  Does
> it really matter or do they all map to the same kernel functionality
> in the end?


If you are using linux and a normal FS (xfs, ext4) then stick with
MD/LVM as it works and is stable and fast.  I thought dm was more
about multipathing and device naming than RAID though?

Another solution to consider is to separate your storage from your
workstation (CPU).  OpenFiler (linux LVM based) or FreeNAS (freebsd
ZFS based) are good for that.  Lets you export NFS, CIFS, AFP, iSCSI
from one nas like box.  Just don't starve them (especially ZFS) for
memory.




More information about the tfug mailing list