Chief Engineer Brian Drown Explains Why Buying a Big SAN Wasn't the Answer

When the producers of Better Call Saul came calling, they posed an interesting challenge for creative finishing house Keep Me Posted (KMP). Better Call Saul would break with tradition in a number of ways. Unlike its predecessor series, Breaking Bad, which was shot on 35mm film, Better Call Saul was shot at 6K on Red Epic Dragon cameras. That meant KMP (a subsidiary of FotoKem) would be working on 4K UHD files. So far, so good.

But Better Call Saul was going to work on a non-traditional post-production schedule. The producers weren't going to move in chronological order, locking and finishing an episode at a time. They wanted to keep all of their media available and open for changes during the process, in case an editorial decision for episode 10, for instance, would benefit from being deliberately set up in one of the earlier weeks.

It posed an interesting thought problem for KMP Chief Engineer Brian Drown. "Better Call Saul presented us with the challenge of keeping their entire season of source media for 10 episodes available through all of online and color," he recalls. It was an unusual request, but it simplified things greatly for producer Diane Mercer, who wanted her editors working freely without the added concern about which media was online and which was offline at any point. And the solution also needed to be affordable, so KMP had to be careful to meet the show's needs without pointlessly overbuilding infrastructure.

KMP set up an SSD SAN tailored to the show's demands. Not a hybrid system, with high-bandwidth SSD alongside slower spinning-disk storage, but an all-SSD system. The SSD system was smaller in capacity than a traditional SAN, but it had the throughput KMP needed to serve the online bay, the final color bay, and to handle transcodes for both dailies and deliverables. 

Would it have saved money to keep at least some of the source material on less-expensive storage? Sure, but that would have been a compromise for a show that avoided compromise. "They were finishing episode 10, and episode 1 hadn't delivered, so they may have been injecting VFX fixes or changing the status of the cut," Drown explains. "That is very non-traditional, and we had to keep all 10 of those episodes live and ready to go at any time. We couldn't have a situation where we were on hybrid or online/nearline storage when we needed to push media back very quickly. Each episode could be 3 to 5 TB, or even a little bit more, so it just wasn't fast enough. The only true solution was to have all 10 episodes sitting on high-performance storage and ready to go at any time."

If the cost of SSD storage meant capacity came at a premium, how did KMP manage that extra expense? It made absolutely certain that multiple copies of the media were not being generated as it moved from place to place in the workflow. Instead, the source footage was copied to the SAN one time, and those files were referenced later in the process instead of being regenerated and passed along at different stages. 

"We implemented what's called a publishing workflow," Drown says. "In a traditional workflow, when the online bay was finished, it would have to push or export new frames to the SAN, therefore eating up more space, whether they were going to VFX or color. We couldn't eat up more space because the UHD frame size is so large. So when we did the export from the online system, we published frames that were just a couple of kilobytes in size and pointed back to the original source media. The new sequences going into color weren't full image sequences, but 'fake frames' that relinked to the originals."

In turn, color might need to export frames for titling. Once again, those "frames" would actually be pointers referencing the original media rather than new sequences taking up precious storage space. VFX, also at KMP, used the same kind of architecture to refer back to the original source files. After the workflow was tested in a proof-of-concept phase, KMP drew up a multipage document defining and describing it, and took the time to train the online editor, colorist, and encoders up front to ensure smooth sailing once the show went online.

Asked if the pointer files ever caused problems as multiple users were redirected to hit the original footage simultaneously, Drown said the team experienced no dropped frames or other performance slowdowns. Again, the squeeze was related to overall capacity, not performance. "It's pretty incredible how much throughput we have off that SAN," he said. "It's unbelievable. But while we had all 10 episodes of Better Call Saul online, we did have to squeeze in the rest of our UHD jobs, and capacity was the issue with the SSD drives."

How big were the savings? Drown says doing it the traditional way would have required networked storage to balloon over 200 TB. But Better Call Saul remained under 80 TB.

The moral of the story is that throwing money at a problem can get a job done, but a carefully tailored workflow can get that job done efficiently. As Drown puts it, "The key isn't that we bought a big SAN. We didn't buy a big SAN. Spending a million dollars wasn't going to get us anywhere, so we spent a very small percentage of that."