Sports broadcast is one of the most challenging tasks in the media industry, as evidenced by the monumental undertaking currently underway as the 2018 World Cup plays out at 12 venues in 11 Russian cities. Live production is, of course, key to the worldwide viewing experience, but the month-long World Cup generates an enormous amount of footage — a reported total of 98,087 broadcast hours in 2014. Much of that footage needs to be stored, both for retrieval during the event by editors looking to cut together highlight reels and other content, but also for posterity, so that broadcasters of the 2022 World Cup have the opportunity to call back to footage from this year’s games when assembling their own programming. StudioDaily spoke to Spectra Logic CTO Matthew T. Starr about the storage issues facing an event like the World Cup, including the affordability of different storage tiers, the future of sports broadcast at 8K and beyond, and why metadata is still king.
StudioDaily: Typically, will the World Cup broadcasters be keeping everything online, or choosing some material to tier down to lower-availability storage?
Matthew T. Starr: They’re probably keeping the primary cameras online, and putting everything else on a second tier. If it didn’t play to air, didn’t play at the truck and get broadcast during the game — for example, the overhead camera — it is probably just going to tier two, where it can be used later on for analysis and pieces like that, while the primary cameras are kept online for at least the first sets of rounds. But then that will probably tier down to at least a stage-two disk, and possibly even off to tape. The Olympics made local tape copies and then sent the tapes back. They were essentially sneakerneting vast amounts of data back to headquarters rather than sending it over the wire, because they wanted to keep every shot and camera angle they had.
And tape is still the default for that kind of cost-per-gig — or, these days, cost-per-petabyte calculation.
The other thing that’s happened in the tape world is LTFS, the linear tape file system. A field system could be based on one type of capture and automation equipment, and then someplace else in the world you’re ingesting it with something totally different on the home office side, and LTFS happens to be the interchange format. It used to be you stuck with one ecosystem out in the field and back at headquarters to keep commonality. LTFS opened that up so it’s almost like a film workflow — you’re writing in an open format as opposed to a proprietary format.
How do they make the calculation of what needs to be stored where, both at the event and over the lifetime of the footage?
The primary job during the World Cup is to put the games on air. But then there’s that at halftime, pre-game and at the end of the game, and late into the evening where they also need to fill some time. That’s where the different storage tiers start to play in. All of the data is being captured and kept somewhere, most likely in Red Square, until they decide what to do with that data — keep it all, throw some away, or purge it to only have what’s played. Most likely, the easiest thing to do, especially if you have the right tiers of storage, is keep it all. During the Mexico-Germany game this weekend, the German center, Müller, got very frustrated at the end of the game. He wasn’t able to score, and he is a scoring monster. And now you can come back and watch his goal-scoring from four years ago and eight years ago and the differences in those games. That’s the kind of data they need to be pulling up from an archive. You need metadata tagging, and you need a deep archive, so you need all those tiers of storage.
What about that archive? Do they have footage from four years ago on site?
It’s a little bit of both. The Olympics took some footage with them last time, and they actually broadcast some out of New York. When they’re building up past highlight reels or comparing athletes’ previous performance to current performance, they will probably do that out of the primary archive.
So they have to have an idea, going into the event, what the key stories are going to be so they have good coverage.
I don’t think Tunisia is going to make the finals, so do I need to bring footage from Tunisia? Probably not. But I’m bringing footage from Spain, Germany, Brazil, Portugal, and England. There will be some Cinderella teams, too, so they’ll still have to pull some material from the mothership, but they want to bring some footage along to help generate more content.
What are some of the challenges of sports broadcast in 2018? You mentioned metadata, and that’s key to actually using stuff in an archive rather than fumbling around for clips.
I always go to metadata. My rule for metadata is there’s never too much. During the game, you’re capturing metadata about who’s on the field, where the ball is, and what action is happening. Should you have information about the angle of the sun or the cloudiness of the sky in the metadata tags for a camera? In theory you could or should. Let’s say you have 15 cameras around a stadium, and you know the angle of the sun in relation to the position of the camera, so you know there will be times the shot will have way too much light or be way too dark just based on the shape of the stadium and where the ball is positioned. Through some automation, when you go to cut a highlight reel, maybe you can rule out shots right away just by having that extra data. So metadata can help you throw out data. Ask yourself, do I care about non-play time? When there’s a penalty and the referee is on the sidelines watching video review, that’s about three minutes of game time. If you’ve got 20 cameras shooting for three minutes, that’s 60 minutes of footage we’re about to drop into an archive, and metadata may be able to tell us to get rid of it. When you talk about metadata, it’s not only what you can drive from a search perspective, but what you can drive later to help automate the editorial process by reducing the amount of footage the editor has to preview for the highlight reel. It’s going to save him time and produce more in the end.
We’re acquiring now in UHD, and these broadcast feeds are available in HDR, but the majority of the audience is still watching on HD channels. Is that a storage issue? Or can the transforms happen in real time so that you avoid storing multiple copies of all this stuff?
I was just in Australia working with Deluxe Digital on an HDR project. Their big issue was: what do they store? Do they store both SDR and HDR? It’s up to them, but my general rule is if recompute is possible and cheap enough, recompute and retranscode. That’s not always the case. But, theoretically, you really just need the original in the highest-definition, best version possible and you can build whatever else you need out of that. If I’m putting six copies in six different formats in a 10-year archive, I’ve probably made a mistake. As technology moves forward, you will be able to do recomputes even more easily.
What about those shots that aren’t unusable, but require a layer of color-correction to match the rest of the footage? Do you keep both versions of those shots?
Let’s say you have a dark shot followed by a light shot, so you lighten up that second scene to match the first clip. I wouldn’t put that [color correction] in the archive. But I may store those automation steps so that I have the automation already done. Just keep the raw content as raw as possible, and later you can come back and transform it.
With the massive amounts of data we’re generating, especially in 4K, anything you can do to reduce how much raw material you have to store without compromising your ability to create valuable content after the fact is gonna have an effect on the bottom line of the production and make it more profitable.
There’s an important piece: valuable content. And what’s not valuable now may become very valuable in the future. The value of content will change over time. Some goes up, some goes way down.
So the trick is anticipating those changes in a programmatic way that hopefully doesn’t require someone sitting there and thinking about every piece of content from every camera.
If you can keep one copy of the game cameras, or three primary angles, stored forever on a cheap enough storage system with metadata that will help you find a player who shot a goal on that goalie eight years ago, that’s the important part here. If you’re going to store it, store it cheap enough that it doesn’t cost your company too much to keep all this content around. And secondly, you have to be able to monetize that content over a long period of time.
We touched on the issue of tape earlier, but spinning disk becomes more competitive all the time. How much are you seeing tape archives being obviated by spinning disk storage?
Let’s go back five or six years. If you had a PB of data, you’d want a tape archive. Today, if you had a PB of data, I’d tell you to put that on disk in an object store. If you had 10 PB of data today, I could tell you that you could probably put it on disk, but you may be able to save a little money by putting it on tape, if you’re OK with the longer access times. But if you have 100 PB of data, you better have a tape archive or you’re going out of business. You’d be spending too much money on your storage platform. The barrier to entry for a tape archive is 10 PB. The barrier for object storage is maybe 1 PB, and online file systems is everything below that — just buy some Isilon storage or similar. At Spectra, we like object storage for a Tier 2 nearline or online system, and we put disk and tape behind our object store. If you had a PB or two and said, “I need fast access,” I would point you toward an object storage archive. But if you said, “I have a PB or two of old news footage, historical content from a public broadcaster that we’re probably never going to touch but need due to regulations,” I would point you to a tape archive. It depends on what your usage patterns are. Both of them solve the problem, but they have a different cost point, so you have to pick the right one based on that.
The broadcast industry is getting used to 4K as the master format for an event like this, but 8K will be another jump, maybe starting in 2022. Does that make the job of building out storage more difficult? Or is it just more capacity?
Once you go to 8K, you need to change all of the parts. It’s not just how much storage you have, but the bandwidth to that storage has to increase to meet the data-rate requirements. So we need to increase both capacity and bandwidth, and we need to look at metadata even more. One of the values of 8K is that you can cut a 4K shot out of it. When you think of those really large, high-density images, you can set a camera back — or maybe two or three cameras — to get the entire playing field and then dynamically move the screen around to show the game. That requires no camera operators. Just put a chip in the ball and have the camera follow it. There’s a device called a wide-area motion imaging (WAMI) sensor that sits on the bottom of an airborne platform that flies over Afghanistan and films 10s of square kilometers at a time. They fly this aircraft and they’re looking for change. They don’t really care about the footage — they care about change in the footage from day to day and week to week. Is someone building a wall or digging a hole, or is there a new truck in the middle of the desert they have to take a look at? But the point is, the wider they get that sensor, the fewer passes they have to make. So those sensors exist, but they’re too expensive right now for a broadcast environment. That doesn’t mean the price isn’t going to come down.
Spectra Logic: spectralogic.com
