Hard Drive Storage for Producers

At least once a week I’m asked, "what hard drive should I buy?" A
bewildering number of options make choosing dedicated storage for media
files seem like a tough question. For a facility that needs a massive
RAID system or storage area network to be shared by several
workstations, it is. But for a producer or editor who just needs
support for common SD and compressed HD formats on a single
workstation, it isn’t. Once you know your throughput, capacity and
redundancy needs, your choices become clear. Well, clearer.

Determine your throughput needs not by the format your camera records
to tape (or card or optical disc), but by the highest-bandwidth format
you regularly load onto your hard drives for general editing. If you
shoot and edit DV, then you’re looking at 3.7 MB/sec. But if you
transcode from HDV to DVCPRO HD, for example, base your calculations on
the 14.8 MB/sec required for DVCPRO HD. For HDCAM, calculate based on
whatever intermediate codec you use. Avid DNxHD, CineForm,
uncompressed, 4:1 MJPEG and other options are all intermediate codecs,
after all.

I then take the highest data rate I’ll regularly encounter and multiply
by four. That will support reading and writing of a couple of video
streams, a few channels of audio (actualities, VO, music), plus a
graphics stream and overhead. Motion-graphics designers working with
uncompressed or lightly compressed material may not be able to find a
system that offers four times their format’s data rate. But if they can
live with, or at least tolerate, non-real-time performance, there are
still affordable options.

If you only occasionally online with uncompressed HD, and usually do
that at a facility other than your own (have you seen the price of HD
scopes?), this approach may work for you.

Video format (and network) data transfer rates are typically described
as megabits per second (Mbps). Drive transfer rates are typically
described in megabytes per second (MBps). Converting between the two is
simple: Eight megabits equal one megabyte. The chart, on page 35,
suggests that if you mainly edit in DV, a storage system that can
sustain 15 MBps will do. If you edit DVCPRO HD, 60 MBps will keep you
happy. Different intermediate codecs have different data-rate needs,
but 120 MBps will keep a few streams of most formats humming along.

Just as every problem expands to fill the available time, so our video expands to fill all available hard drive space.

The size of a storage system depends on your editing format’s data
rate, the amount of material required for a typical project and the
number of projects that will simultaneously be on the device.

I have yet to encounter a typical project, but I have found some
typical source-to-final ratios. This isn’t the same as a shooting
ratio. I reduce the effects of location imagorrhea
with logging to trim down what ends up on hard drives.

For fiction and corporate work, 5:1 is a good benchmark for rough
footage, graphics, render files and so on. For a documentary, the ratio
may increase to 10:1 or 20:1. A motion graphic designer may have a
higher source-to-final ratio than a documentary or corporate editor,
but will also be working on shorter projects. So let’s call it a wash.

While a five-minute corporate piece shot and edited in DV may require
only five gigabytes, a one-hour documentary edited in DVCPRO HD could
require over a terabyte.

I work on multiple projects at the same time, usually one to three
corporate jobs and one to three documentaries or short dramas. There’s
the job I’m finishing, the job I’m working on and the job I’m sitting
on while the client (and for the docs, that’s usually me) figures out
what the heck they’re going to do. My corporate work is mostly standard
definition (though that’s changing); the docs vary from DV to HDCAM
originals with editing taking place in some format that consumes less
than 75 GB per hour.

For me, one to three terabytes of mainline storage does the trick. And,
in fact, each of my two main workstations (one Mac and one Windows) has
a terabyte of main drive space for media files.

For video, the five leading computer-to-storage interfaces listed in
general ascending order of performance are FireWire 400 (i.e., standard
FireWire as found on DV cameras), FireWire 800, SATA, UltraSCSI and
Fibre Channel. I’m mixing technologies here. Typical "FireWire" drives
have either IDE or SATA mechanisms inside. Similarly, there are arrays
built around IDE drives, but with SCSI interfaces to the computer. But
most users group drives based on the type of cable they plug into the
device.

For modern video formats, FireWire 400 is getting long in the tooth and
slow on the track. Even with the recent Oxford Semiconductor 924
interface and a modern 500 GB SATA II drive, the best read performance
I’ve seen with FireWire 400 was just over 30 MBps. That’s good for over
four streams of DV and four of DVCPRO50, but not fast enough for
reliable and efficient use with most other formats. I’ve seen Firewire
400 drives alternately play back HDV footage without a problem, but
then drop HDV frames. However, I’ve always been able to reliably play
back the same HDV footage on an internal SATA drive I’d investigate
further if there weren’t better alternatives. But there are. For
example, an external drive built around the Oxford 924 can offer
FireWire 400 and 800, USB 2.0 and SATA ports. A single drive can give
you good performance at home and great compatibility on the road.

FireWire 800 offers close to twice the throughput of FireWire 400 and
is viable with more formats, as long as you have an independent
FireWire bus that isn’t also supporting a VTR or camera. On most
computers all FireWire ports use a single bus, and occasionally a
camera’s or VTR’s FireWire signal will conflict with that of a FireWire
drive. You can avoid the conflicts by installing a second FireWire bus
through a PCI card in your desktop or a PC card or Express card in your
laptop computer.

But if you don’t already have a spare FireWire 800 port, you could go
to SATA. Big moderate-priced drives can sustain around 70 MBps, and you
can combine drives for faster performance. You can get SATA cards for
desktop and laptops. For my needs, SATA is the way to go. At least for
this year. I don’t have anything against FireWire 400 drives. I have
six right here. While I don’t use them for main online storage, I use
them for backup, some location work and shipping media to collaborators.

It comes down to speed, data security and cost. Two or more drives set
up as a RAID 0, with files spread evenly across the drives, gives great
performance at a low cost. For example, a single SATA drive that can
transfer around 65 MBps (and that slows down as it gets full) could be
matched with three similar drives into a RAID 0 array and transfer over
200 MBps.

But if any one of the drive mechanisms in a RAID 0 fails, data on all
of them is lost. RAID levels 3, 5 and 6 add data security and
redundancy by adding increasingly secure parity information needed to
recreate files across drives. With RAID 6, you can theoretically have
two drives fail and still not lose data.

There are other RAID levels, and each level has drawbacks and
advantages. But overall, this data security often comes at the cost of
reduced throughput and storage capacity, and always comes at the cost
of increased, well, cost.

A 2-terabyte RAID 0 can cost $2,000. A 2-terabyte Fiber Channel RAID 3
or 5 system appropriate for video work can easily cost $4,000 to $7,000
or more. If you run a full-time facility with multiple seats and
multiple editors, then a higher-level RAID system or storage area
network is a good economic choice. Lost time is lots of lost money.

But my two workstations each have their own RAID 0 system, and I
regularly back up the data on each to FireWire drives. It’s not a
perfect solution and I plan to buy a higher-end RAID system in 2007,
but for now it works.

I used to track storage technology more closely than I do now. But
knowing the best mechanism of yesteryear doesn’t mean I know the best
choices today. Western Digital and Maxtor used to be on my do-not-fly
list. Now some people have great experiences with them.

Tracking storage trends and building systems can be fun. But I have
other hobbies, so these days I buy from video-savvy vendors. It saves
time, doesn’t cost much extra and gives me a direct line to tech
support and replacement parts if things go wrong (which they haven’t,
knock on wood).

I don’t buy the lowest cost-per-gigabyte systems available at Best Buy.
Video pros work their drives harder than consumer and business users
do, and our systems, therefore, generate more heat. So even if the guy
at the megastore says you can stick four more hard drives into your
computer, or says he edits video with the FireWire drive that’s on sale
today, you might not get what you need.

Many of the storage questions I get come from people whose FireWire
drive just went kaput. It often turns out they bought a consumer
FireWire drive that didn’t have an internal fan or heat sink adequate
to keep cool a constantly working drive. And I’ve seen similar problems
with some RAID systems built into computer cases.

But I’ve also seen reliable fanless FireWire drives and successful
internal RAIDs. The difference is intelligent heat dissipation, and I
find that intelligence most reliably with vendors and resellers who
really understand the needs of video professionals.

Storage is cheap. Figure a buck or three a gigabyte. Or in video-sized
terms, around $1,000 per terabyte for single drives and SATA RAID 0,
around $3,000 per terabyte for multi-terabyte Fibre Channel systems.
Compared to the guy in 1997 who rented out 9 GB hard drives for $125
per week, hard drive storage today is a bargain.

– Tips for Implementing Storage

– Underdog editor Brett Schlaman: Why Archion RAID Takes a Bite Out of Time on an Effects-Heavy Avid Edit