[Tfug] Small-ish (capacity + size) disk alternatives

[Bexley Hall]

Hi Bender,

On 2/3/2013 10:58 AM, Bender wrote:
> LOL - a Bexley boondoggle..?
> What follows below is my reply to this thread back on 1/30. But after
> seeing threads like one this turn into sticky goo previously, I
> reconsidered then.
>
> WRT to Tyler, John, Nathan and the others -- Is it as though he already
> knows the answer? If so, why does he engage in this? As one who tells us
> he does this professionally, is he just stirring the pot, or what?

I design embedded systems, professionally (things that are obviously
"computers" yet don't look like conventional computers).  To date,
none with a "conventional" hard disk drive.  It would be foolish *NOT*
to ask about an area with which I have no experience!  Especially
*laptop* drives (since I rarely even *use* a laptop).

I.e., my post could easily have been met with comments like "laptop
drives are notoriously low reliability!  I replace mine every year".
Or, "avoid drives made by Finklegruber".  Or, "this sort of problem
plagues the smaller drives -- try something larger".  But, since it
wasn't, it suggests that others have no direct experience using
laptop drives in a similar situation and can't comment on that,
directly.

>> My home automation/multimedia system relies on a single
>> smallish (20-30G) disk for "read only" (i.e., binaries)
>> storage as well as "bulk" read-write memory (i.e., 30G
>> of RAM would be costly, bulky and power hungry -- and
>> far faster than necessary).
>>
>> Currently, I've been using small laptop drives. They
>> don't see much use (thrashing, etc.) but are spinning
>> 365/24/7.
>>
>> It appears this is not the way they like to operate
>> (i.e., they seem to die pretty easily). Heat shouldn't
>> be a problem -- enclosures are cool and well ventilated.
>> I'm just guessing they don't like running "forever"
>> with no chance to spin down (or off).
>>
>> So, what alternatives do I have to replace them?
>
> You could also rethink your architecture choices.

There's nothing wrong with the "architecture" -- just this
initial choice of R/W storage medium.  The only other architectural
change would be to centralize *everything* and have a HUGE machine
that had to do everything (along with all the required I/O's) in
one place (which poses even more reliability issues as now
everything is in the critical path instead of just a disk drive).

> Ostensibly, this platform has outscaled its initial parameters.

I don't see how you can reach that conclusion.  What resource(s)
have *grown* to surpass their initial requirements?

> With what you have you could set it to sleep during periods of non/low-use.

My first (of 3) laptop disk died sitting in a laptop.  OK, laptop is
manhandled more than any other bit of kit, drive had some years on it,
lack of ventilation, heat, etc.

My second died in a 365/24/7 system with no severe environmental nor
usage threats.  I *guessed* it was because the drive was going to sleep
and, almost immediately thereafter, being called to spin up, again
(since the drive was simply sleeping after N minutes of inactivity
and periodic tasks just happened to come along almost as frequently).

When deploying for this new application, I opted NOT to let the drive
sleep hoping the frequent spin-up was the stressor.  But, I didn't
get any better results with that approach, either.

So, from a TINY SAMPLE, I suspected laptop drives just won't cut it
(because they are intended to be deployed in a product with a relatively
short lifespan?)

Note, however, that only one of these examples is the "typical"
laptop application -- the other two could be seen as misapplications
of the technology (so, wrong to blame the technology if it simply
isn't fit for the task!)

> But for home automation, you might not care to wait 5 seconds for the disk
> to spin up and then wait for it to wake up.

You're not understanding how the drive is used.  It *won't* sleep.
Ever!  Just like RAM doesn't sleep (i.e., it is used as backing
store for VM).  As the database is queried (continuously by the various
clients that implement the system), the disk acts to hold the temporary
tables created by those queries (which will typically be deleted
within seconds of the query finishing) as well as "overflow" memory
that the RDBMS (and OS) need to satisfy those requests.

> If the SSD idea doesn't peak, it's time to ditch the high level software
> and go low level software and/or specialized hardware.

My experience with SSD's comes second hands from colleagues who have
tried (unsuccessfully) to adopt them (in products, laptops and data
centers).  Since those colleagues work in similar domains as I, we've
tried to analyze each "failure" by looking at the underlying
technology -- not just reading articles in magazines, etc.  I.e.,
canvas the sorts of FLASH devices ("chips") that are available.
Make educated guesses as to which sorts of devices are actually being
used (the endurance factors vary significantly) along with other
architectural design choices (how many controllers, etc.).  Then,
look at what the SSD is *actually* seeing in terms of use (you might
*think* you are writing X bytes but the SSD is actually having to
deal with an entirely different workload!).

In almost every case, back-of-the-napkin calculations are close to
what has been *observed* (by whomever was complaining about the
poor performance!).  And, the consequences of the common remedies
(e.g., caching, overprovisioning explicitly or behind-the-scenes,
etc.) also fall right out of those calculations.  ("Gee, if you had
more NAND hidden away someplace, you could get greater life!  Sort
of like keeping a spare roll of toilet paper 'hidden' behind the
one in use...").  I.e., it's a deception, not really an improvement
in the technology!  (My friend's truck can travel more than 1000 miles
on a single tank of fuel!  But, he's got 100 gallon tank on his truck!!)

So, when I went in this direction (on other project), I spent a fair
bit of time understanding what was going to happen behind-the-scenes
so that I wasn't surprised by the level of activity that the storage
system saw.  E.g., DBMS effectively writes all data *twice* (once for
real and once for the write-ahead-log).

You'd never be able to code all of this in a lower level implementation
in a calendar lifetime!  And, it is, undoubtedly, the way such systems
will continue to evolve in the future.  E.g., manufacturers already have
designs for "smart appliances" -- but no demand for them (where is the
home automation system that can tie them together?)  You can purchase
"audio" distribution systems and a slew of other stand-alone appliances
(some which have connectivity options/capabilitites).  But, as yet,
no way for them to all talk together (and no one to coordinate those
"discussions").

E.g., in the future, the electric company will offer tariffs that
allow them to dynamically *shed* load in individual residences to
better manage their power distribution.  I.e., turn OFF your air
conditioner compressor for a while -- along with some of your
neighbors, etc. -- to accommodate a sudden peak demand for power
that they *can't* control.  Easier/cheaper for them to do this
(by offering you a financial incentive for this "sacrifice") than
to build another power plant.  Ditto with your refrigerator and
electric vehicle (imagine everyone coming home from work at 5:00PM
and plugging in their cars for a recharge at the same time??).
The same applies to natural gas (utilities want to see a *steady*
load, not one that has large variations over time).

[Large businesses pay for their power using different tariffs.
E.g., based on their *peak* power consumption.  So, you'll see
wacky systems in place that do things like "make ice" at night
so their air conditioning systems don't have to consume as much
energy during the day (obviously, it is inefficient to make ice
instead of just cooling the air!)]

Someone (with much deeper pockets than me) will design a mass-marketed,
low power, bulk storage system (e.g., like Apple's "reduced performance"
disk drives used in their early iPods; or, something akin to the old
pseudo-static RAM that could easily be idled without a controller).

For now, I'll try the drives that came yesterday and see how they
fare a year from now (or not -- if they crap out even quicker!).
And, possibly look into a physical RAMdisk (the only one I've
seen aim for very high performance which is the wrong direction
on the price-performance curve).

Or, just hope some other technology with a "disk interface" comes
along...