Minding the Store

In real estate it’s said that the three keys to success are location, location, location. For video production professionals dealing with high-definition video and 2K and 4K resolutions now handled as digital data, that adage could easily become "storage, storage and more storage."

The biggest challenge is picking the right capacity for your project and the most robust technology to ensure your stored data is secure. After that, it comes down to two key considerations: the data rate you’ll be working at (uncompressed or compressed) and the amount of raw footage you’ll be dealing with.

Will you be setting up a direct-attached storage (DAS) array for a single user, network attached storage (NAS) for dozens of editors, or a sophisticated storage area network (SAN) for hundreds of users? Drive arrays should also incorporate some type of cooling system because video editing puts a lot of strain on the drives during the editing process.

When picking a storage system, it’s important to remember that there are very affordable 400 and 500 GB drives (about $400-$600) available now, so capacity these days is a non-issue. Bandwidth (getting data on and off the drives) is everything and determines what you can and can’t do.

For compressed HD projects, you’ll need about 35 to 40 Mbps throughput if you’re using material compressed with DVCPRO HD, HDV, Apple’s 422 codec or Avid’s DNxHD codec. That’s also what you’ll need to do uncompressed SD projects (compressed DV requires a mere 4 to 5 Mbps). Uncompressed editing and non-real-time effects require a minimum bandwidth of 150 Mbps, and even higher for real-time effects work.

If you’re putting in new storage, manufacturers recommend that you install a second FireWire card in your computer so you have a clean bus that’s not doing anything else but moving material on and off the drives. It makes for a more reliable video stream coming from the drive.

The most commonly used formats for storage protection today are RAID 3 and RAID 5 disk arrays, made up of 2 to 24 drives, which protect against any one disk failing. Should a disk fail to respond properly, the RAID technology automatically moves the data over to a second disk and the user can continue to work. With RAID 6 technology, you can lose up to two disks in the array and the data remains secure.

Of course, there are tradeoffs with each type of RAID protection. RAID 0, for example, gives you fast access to material stored on the drives, but it’s risky: If you lose a drive, you lose your data. Many single-user environments employ this technology because it’s less expensive and lets you work faster. RAID 1 provides protection by writing the data to two drives simultaneously. This way, you always have the original material securely stored, though it can be expensive, since you’re using half your disk capacity for protection.

"There are some people who roll the dice with RAID 0 because you get a bit more capacity and throughput," says Pete Schlatter, director of marketing for G-Technology, Inc. Your systems run faster, he says, because you’re not using disk space for parity information, which provides redundancy.

G-Technology storage is available in a number of configurations (using Hitachi hard drives), from single disk, bus-powered FireWire drives for use with laptops, up to sixteen 1 TB drives in a rack-mounted Fibre Channel disk drive array that’s only 4 RU tall. Fibre Channel is recommended for multi-user post environments working off a switched SAN.

The company’s most popular product is the G-RAID, which offers SD and compressed HD users fast FireWire (Mac) and USB (PC) connectivity and a two-drive RAID 0 array that offers up to 2 TB of space. (Many editors, incidentally, will tell you that FireWire performs better than USB.) The system is designed for single users working on Apple’s Final Cut Pro or Adobe Systems Premiere NLE systems.

For the single workstation and small workgroup user, Avid Technology offers its Avid Unity MediaNetwork 5.0 system- designed for real-time shared storage- with newly designed system architecture. By consolidating several hardware components into a single, integrated file management system and storage server, Avid says the Unity MediaNetwork 5.0 system delivers twice the performance of previous generation Unity systems while maintaining data resiliency.

The Avid Unity MediaNetwork 5.0 system is available in scalable SD and HD configurations, both of which offer Gigabit Ethernet connectivity and 4 Gbps Fibre Channel connectivity, PC and Mac compatibility, as well as storage capacity that can scale from 4 to 40 TB.

EditShare offers shared-storage solutions to enable Apple and Avid NLE software users to collaborate on the same project and access the same data simultaneously. This lets multiple editors have instant read-only access to each other’s bins and sequences. Leveraging Gigabit Ethernet connectivity, EditShare technology supports a "rules-driven" workflow, which ensures that no data is ever overwritten or accidentally destroyed.

EditShare was one of the first collaborative storage systems to offer 10-Gb Ethernet connectivity for multi-stream SD and HD editing. At NAB 2007 the company unveiled a storage concept called ESA (Extreme Scalable Architecture) that allows customers to increase their storage to hundreds of terabytes and beyond. With EditShare ESA, additional "expansion servers" can be plugged into existing EditShare networks to cost effectively expand storage capacity as well as the number of users on the network. Administrators manage all ESA servers through a single unified graphical user interface.

With EditShare 5.0, duplicate media files move across two or more servers- whether they’re located in the same building or on the other side of the world. Available for Avid systems, EditShare Project Sharing technology has also just been extended, in shipping versions of EditShare 5.0, to Apple’s Final Cut Pro. The feature lets multiple editors eliminate the time-consuming task of exporting and importing of XML files and duplication of FCP project files.

Alexander Buono is a producer/cinematographer working on a feature-length documentary called Bigger, Faster, Stronger that examines the parallel between steroids in sports and ethics in American culture. Shot with two Panasonic HVX200 P2 HD camcorders (and Sanken wireless and boom mics) over a year, the producers combined well over 1,000 hours of source material on three Avid Xpress Pro HD NLE systems. This presented them with an enormous storage challenge.

Buono and his team chose a Synergy HD storage system from Archion Technologies, which includes two 8 TB blocks of storage across the HD RAID 3 array. The Synergy HD array is designed to expand the storage requirements of existing Avid Unity clients without customers having to replace or upgrade their current Unity infrastructure.

Archion’s Synergy HD4 is a SATA II RAID storage system, available in units from 4 TB to 12 TB with two ports of 4 GB Fibre Channel storage. The company targets Avid Unity users as its storage array is fully compatible with Unity versions 3.3 through 4.2.2. The new Synergy HD4 extends Archion’s original Synergy HD abilities from 400 to 800 MBps bandwidth with its dual 4 GB FC ports and has increased its capacity from 8 TB to 12 TB in the same 3 RU space.

"We started with a different storage system that wasn’t adequate in terms of bandwidth," says Buono. "It was incapable of dealing with three Avid HD systems at once. At one point we lost 2 TB of data, but we were able to reload our back-up drives and off we went. It turns out we needed a lot more storage than we originally expected. With documentaries, you always end up shooting more footage than you think you will. It’s just something you can’t control."

ATTO Technology markets storage connectivity and infrastructure solutions that provide reliable high-performance connectivity storage arrays for digital editing. The company’s FastStream SC 5300 storage controller, a standalone unit that adds hardware RAID protection to Avid MediaDock Ultra320 storage configurations, lets customers dynamically protect, manage and extend existing high-performance storage systems.

FastStream SC 5300 is a 1 RU box that features two 4 Gbps Fibre Channel host interface ports and two Ultra320 SCSI drive interfaces for native SCSI drive support. FastStream is optimized for Avid editing systems and protects data without compromising system performance. Professionals can edit up to two uncompressed streams of 10-bit HD video with an alpha channel or 11 streams of uncompressed SD video while protecting the data. The FastStream storage controller’s 480 MBps throughput makes it a good fit for production and post environments.

ATTO’s ExpressSAS RAID Adapter family provides SAS/SATA II connectivity and delivers 3 Gbps per port performance for digital media users. They provide advanced fault-tolerant features that include hot swap, hot spares, online capacity expansion, battery back-up and optimized disk utilization for SAS/SATA or SATA II drives.

The adapters support standard RAID levels 0, 1, 4, 5, 6, 10, 50 and 60 and feature ATTO’s exclusive DVRAID, making them capable of handling the most demanding applications such as 2K and 4K film, multiple layers of complex, uncompressed real-time effects and HD video editing.

Ciprico has been one of the pioneers of "video centric" storage, with DAS, NAS and SAN products targeting content creators, from DV25 projects to high-end applications for 2K and 4K applications. The company’s rack-mountable MediaVault U320RX, for example, uses parallel ATA drive technology to provide up to 5 TB of cost-effective, "video-tuned" storage that can be configured for RAID 0 or RAID 3 operation. It can be configured with up to 10 removable drives, two removable 300-watt power supplies and a dual-channel Ultra 320 interface to provide throughput speeds in excess of 400 MBps. That’s enough to support real-time, 10-bit HD applications commonly used in editing applications.

Ciprico also offers a new DAS system using the new PCI Express external cabling standard (which is also used by Sony’s new XDCAM EX camcorder). Based on the company’s RAIDCore software RAID stack, the new system lets users add capacity within a single cell, creating a storage pod of up to 64 TB. The combination of a performance-switched DAS cell with a more scalable SAN-based network provides a cost-effective, secure, two-tier scalable storage environment.

DataDirect Networks markets NAS, SAN and tiered storage solutions as part of its Silicon Storage Appliance (S2A) product family. At the NAB convention, DataDirect showed a NAS solution, an HPC storage solution, an Active Archive and Nearline solution, and the company’s S2A9550 system that uses a CXFS SAN file system. All are designed to fit the requirements of broadcast operations that need fast access to the data among a number of users on a network.

iQstor Networks, Inc., offers what it calls "intelligent" SAN solutions for small and medium production companies. Products include the iQstor iQ2880, a 4 Gbps Fibre Channel storage system with volume management-based virtualization, snapshot, mirroring, remote replication, storage provisioning and scalability. The iQstor iQ2880 can be scaled using the J2880 SBOD expansion enclosure, creating a system with high-performance FC drives for primary storage and low-cost SATA drives for secondary storage.

The system’s drive canisters include aN FC-SATA bridge module, which means high-capacity SATA2 drives can be used with FC drives within the same enclosure. Each iQ2880 supports up to 15 drives, providing 4.5 TB of storage capacity using 300 GB FC4 or 11.25 TB using 750 GB SATA2 drives, and can scale to 72 TB with FC or 180 TB with SATA2 drives, offering high-capacity and increased throughput.

Larger, more sophisticated storage systems utilizing dual-core Intel Xeon processors that are linked to serial and parallel ATA storage drives- the difference is in how the data is stored and accessed across the drives- have also become popular with big organizations handling massive amounts of data in their daily productions.

SATA is a technology designed for desktop computers that was shunned by many professionals only a few years ago. The difference now is that RAID 5 and RAID 6 technology has the added fault tolerance required by IT managers and large content delivery facilities. RAID-enabled SATA storage is now seen as more than adequate and costs less than the now more commonly used SCSI technology.

Large organizations, such as broadcast networks, with multiple locations and hundreds of users are also beginning to embrace automatically managed storage in a big way. NBC Universal announced that it has chosen a large system from Isilon Systems to archive and access its ever-increasing stores of media programming across its multiple divisions. The network is installing the IT-centric storage systems to replace existing tape-based technology, enabling faster access times and increased reliability.

Under a new multi-year agreement with Isilon, NBC Universal will deploy multiple Petabytes of Isilon IQ clustered storage in its main production and broadcast operations in the United States. NBC is also using the Isilon IQ storage system to support coverage of the 2008 Olympic Games in Beijing. (The network used Isilon IQ at the 2004 Olympic Games in Athens and the 2006 Olympic Winter Games in Torino.)

Next summer in Beijing, NBC, for the first time, will produce and broadcast the Olympics completely in HD. The network will store the majority of the Beijing Games as low-resolution proxy files on Isilon clustered storage, which will give NBC producers fast and easy access to content for rapid review, identification and selection. This will facilitate quick turnaround times on edited segments and let producers add "back story" elements to live coverage.

Powered by highly specialized OneFS software, Isilon IQ is a single file system that unifies and provides instant access to digital content via a clustered storage architecture. Isilon also provides a suite of software applications that leverage OneFS and clustered storage for data protection and automated data management.

In a world of uncompressed HD files that must be stored and accessed off the drives quickly, data security is always a concern. Wren Waters, an online/EFX artist and partner at Therapy Studios, in Santa Monica, CA, says that because his facility often works off the actual HD media from the camera (to preserve quality), the inherent large file sizes of HD create a problem. Therapy Studios uses a 6 TB Avid LAN Share system, but Waters and his colleagues feel that disk drives are not the only option for data storage. Sometimes tape is still the best option.

"In the past we would digitize at DV25 resolution, so your storage needs are much less," he says. "We’re now storing the media in its native DVCPRO HD 720p resolution because it creates the most streamlined workflow. But it also means we’ll need around 3 to 5 TB of storage capacity."

An even bigger concern is how to securely back up this media, which now exists in one central location. "In our experience, having data backed up on a FireWire drive is not the most stable or secure way to handle data for a project like this," says Waters. "And we don’t want to trust our entire investment in this film to FireWire drives. So we’re trying to figure out the best way to aggregate the data."

He says he and his colleagues back up most of their online projects to DLT tape, which provides lots of storage capacity- from 160 GB to 320 GB with compression. "We might end up with about 15 or 20 of those tapes for a particular job, only because, in our minds, that’s a far more stable format to back up our data to," Waters explains. "If you lose your media, it’s gone forever."

For the user, it’s buyer beware. The storage options available are numerous, so always understand first how you like to work before you try to find a system that fits your workflow and your budget.

In its effort to sell signal conversion cards and provide a valuable tool for the industry, AJA Video (www.ajavideo.com) has a very useful (and free for download) software calculator on its Web site that helps video professionals determine the correct amount of storage required for their PC or Mac projects.

Users can adjust numerous parameters (for example, frame size, frame rate, compression, sample rate for audio, bits per sample for audio and number of audio channels) as well as the time (hours, minutes or seconds times a multiplier). This provides a flexible way to approximate storage needs.

On the Mac side, pull down Support, and then select Software Downloads. From the list of products, click on Kona3, then scroll down to AJA Data Rate Calculator Application Version 2 at www.aja.com/html/support_konaLHe_swd.html/.

For PC users, select the Xena card and follow a similar path to AJA Data Rate Calculator Application Version 1 for Windows at www.aja.com/html/support_xenaLHe_swd.html. – M.G.

WWW.ARCHION.COM
WWW.ATTOTECH.COM
WWW.AVID.COM
WWW.CIPRICO.COM
WWW.DATADIRECTNET.COM
WWW.EDITSHARE.COM
WWW.G-TECHNOLOGY.COM
WWW.IQSTOR.COM