How RLTV Kicked the Videotape Habit

For James Peebles, VP of engineering at Retirement Living TV (RLTV), building a new automated, tier-based archiving system from the ground up meant an opportunity to put the finishing touches on concepts that he started developing in 2000, when he worked for Oxygen Cable. As the emergence of workgroup editing created shared storage pools that were exploding in size, Peebles realized that existing data-management systems weren’t up to the task of handling 21st century workflows. “The easy part is hooking up your editing machines and getting them to work together,” he says. “The hard part is managing the data you’re acquiring – and managing it in such a way that you can use it later and organize it sensibly."

The problem, in other words, isn’t just storing footage on ever-larger volumes. You need a strategy that gives your storage system information about itself, and instructions on how to handle all those bits and bytes in a way that computes in terms of bandwidth requirements for backups, flexibility when it comes to ingest and playout, and, last but certainly not least, cost-efficiency. There are lots of ways to build that kind of system, and each one has its ups and downs. But Peeble and his team at RLTV dedicated themselves to building a system that would not just keep up with technological change, but with the cultural change that comes with the arrival of totally tapeless workflows.

“We have a truly file-based workflow, and we’re one of the few networks we know of that has completely eliminated videotape from the process,” Peebles says. “A lot of networks are trying to reuse and repurpose legacy systems, with a production style that is built on videotape with producers and editors programmed to work in that mode. We’ve shaken all that off, retraining our editors and producers to work in an all-digital environment.”

Peebles knew he wanted to move RLTV away from the industry’s videotape legacy, which meant archival capabilities and disaster recovery would be a key issue. The first question had to do with what general components would make up the system – specifically, did it make more sense to go with an all-disk-based system, or to try to integrate tape backups with spinning disks? RLTV took a rigorous look at pure disk-based solutions, noting that companies like Isilon offer well-engineered, expandable systems that allow storage to grow organically as a facility’s needs increase. But Peebles found that the overhead costs of keeping a large spinning-disk system running would be substantial, and wondered about the implementation of a backup system for disaster recovery.

“As you build up more banks of disks, you’re having multiple devices running simultaneously, drawing tremendous amounts of power and creating tremendous amounts of heat. That has a lot of costs,” he explains. “And backing up the system puts you on the horns of a dilemma: if you’re going to stay all spinning disk, you have to replicate that system somewhere else, with a data pipe copying data from one physical location to another, so that you have true redundancy. That’s a very expensive solution that requires a lot of horsepower, a lot of electricity, and a lot of cooling. It’s all moving parts, all the time.”

Peebles decided instead to lean on LTO tape backups, mainly because the media only has to be loaded and spun up to speed when it’s actually being used by the system. And, if you can instruct your storage solution to write everything that goes to data tape onto a second physical tape unit, you can physically transport those tapes to a secure off-site location. RLTV eventually decided to put a small amount of disk-based storage at the front end of the system, with a large data-tape vault on the back end. It wasn’t hard to put the system together, and Peebles approached vendors he had worked with in the past. Apple provided its Xserve and Final Cut Server technology, Quantum provided its StorNext data-management software and the Scalar i2000 tape library, QLogic provided fiber-switching infrastructure and Cisco Ethernet switching, and Promise Technology became the go-to provider as RLTV needed to expand its disk-based storage capacity.

Rock Around the Clock

“The original design intent allowed for two basic things,” says Andrew Richards, RLTV’s director of engineering technology, who worked closely with Peebles to build the system. “One was editing, if necessary, around the clock on multiple workstations, all sharing storage. And the other was to maintain file-based output rather than creating tape masters of the finished content. In order to pull that off, we needed to have a shared-file system that could cycle its data to a nearline platform without impacting the performance of the system to allow editing to go on.” That meant traditional backup techniques, which use up a chunk of bandwidth just to transfer copies of data to multiple tape drives, were out of the question. Instead, the StorNext software looks at the disk-based storage, file by file, as new data is created and replicates the new pieces of data to tape. Essentially, the backup is constantly running – but never using up enough bandwidth to impact the editorial process in multiple rooms.

“It’s an intelligent system,” Peebles says, “unlike traditional backup systems where an operator has to get involved in physically managing your backups and incrementals. We tell it, ‘when you make a copy, make two copies.’ We take one out and put it in a cardboard box in a secure location, and that’s the vault.” Peebles had tried to make similar systems work before, but he says the technology just couldn’t keep up with itself. “When we moved to our first SAN systems at Oxygen Cable, I tried to do backup systems that would work, but the rate at which data was changing was so high that there wasn’t a tape robot in the world that could keep up with a backup.”

Richards explains further. “The metadata of the file system that holds the data drives the policies of moving data to tape. That happens at such a low level that it’s not going to overwhelm the bandwidth of the system the way old-style overnight backups would. It’s just quietly trickling away, keeping the data safe. The policy we have set up will put the data on tape within 15 minutes of its being introduced to the system.

Final Cut Server in the Mix

Final Cut Server came into the picture, two years ago, as an exceptionally functional digital asset management solution. StorNext and FCS don’t exactly talk to each other, Richards says, but they work exceptionally well together. “Part of our build-out for Final Cut Server included the desire to completely blow up our file system and start with a fresh arrangement to complement what FCS could automate for us,” he explains. After the release of Final Cut Server in 2007, RLTV shrank its existing file system to fit on some newly purchased disks, then repurposed the original Apple storage system to build a larger file system from the ground up. “That brought us into a better logical arrangement, with FCS managing the creation of directories for us against projects the teams were starting. It also manages the movement of finished content on and off of nearline storage, based on user input. Before FCS we had to manually interact with the file system just to get data back off of tape for people But now, they’re just a few mouseclicks from recovering content from the nearline tape archives.”

When FCS moves a file to a given location, StorNext takes the ball, moving the file to tape and deleting it from the disk. If FCS comes looking for that file again, it just has to wait for it to be retrieved by the Scalar i2000 library and then put it back online. (One of the best features of the i2000, according to Richards, is its highly predictable cost. “We chose that platform because it could expand into the petabytes if necessary,” he says. “Every time you buy another 100 slots, regardless of whether or not a new cabinet is required to accommodate them, it’s always the same price. That was appealing – we always know how much it will cost to expand, regardless of how full the hardware is.”)

And Final Cut Server turned out to offer extra capabilities that the team at RLTV wasn’t even aware of when it purchased the system. “Because of the way it hooks into Final Cut systems like Compressor and Qmaster, Final Cut Server has become part of our distribution engine,” Peeble explains. That means FCS can be used to transcode video for use on the Internet, mobile phones, and elsewhere. “We run scripts from FCS to create and distribute media by FTP anywhere in the world. It became an opportunity to fully automate the process – when we are responsible for acquisition, we don’t even go to tape. We build spinning-disk-based acquisition systems that take us straight into the system. And if we take acquisition from outside the system, we have enough decks to ingest it once, and it never touches videotape again. It’s all file-based from then on, for ever after.”

Set It and Forget It?

It took a lot of work in the planning stages, but now that the system is in place, it requires remarkably little care. “It took us a while to get it running perfectly, but now that it’s running it takes care of itself,” Peebles says. “We even had a catastrophic event – we were in a temporary location where the air-conditioning system failed over a weekend and cooked the whole system to 120 degrees. It corrupted the system, which we had to blow up and rebuild. So we actually proved that disaster recovery works – the hard way.”

“This was a file system of 25 TB,” Richards recalls. There was downtime to actually recover the file system, but the data was intact.”

The main production file system at RLTV today is 35 TB, with an additional 10 TB set aside for swap files and proxy storage. The Scalar i2000 is licensed to hold up to 85 TB, and the tape vault holds about 200 TB worth of data.

One of the reasons this is all so affordable is because RLTV is standardized around DV25, which is still standard definition, but can be easily run through a Teranex converter that will make it look exceptionally good at HD resolutions. “We’re poised to make a transition to ProRes or DV100, but we’re still DV25 until there’s a demand from our distribution side to convert,” Peebles says, noting that the system is integrated with playout. “Master Control plugs into our storage system. We move our finished content into our playout system, segment it, and play it to air without any tape intervention at all. Frankly, we’re capable of moving it all to the Internet if we want. Because we’re fairly new as a company, we have been able to build this kind of system from the ground up.”

As companies grapple with the HD transition, Richards suggests they should consider making a break with videotape-based tradition. “If they’re going HD, they can go file-based at the same time and shrug off a lot of old habits and techniques,” he says. “Culturally, there can be a lot to overcome. There’s industrial inertia from people who have been doing this for decades. But the cost advantages are so significant that it’s impossible to ignore going to a file-based technique like this.”