Now That’s a Changeup: MLB Network’s Offsite Infrastructure

One of the most daunting challenges in the media industry is media management for a major sports organization. Think about the amount of content flowing in on a daily basis during the season, when as many as 15 Major League Baseball games are being played on a given day — each of them defined by a series of pitches and at-bats that demand a wealth of metadata description — and then consider the archival requirements as every one of those pitches must be maintained for posterity and easy retrieval. And then consider that MLB Network is itself a broadcaster, with programming dedicated to baseball. Keeping that beast fed with up-to-the-minute sports stats and video highlights would be a challenge all by itself. MLB keeps up with a custom asset-management infrastructure it calls DIAMOND, a backronym for Digitized Industry Assets Managed Optimally for Networked Distribution.

DIAMOND has been around for a while, but it's only in the last few years that much of its infrastructure has been moved offsite to a data center near its Secaucus, NJ headquarters. How does it all work? Tab Butler, senior director of media management and post-production for MLB Network, spoke at the Association of Moving Image Archivists' Digital Asset Symposium, held this week in New York City. Here are some highlights from his talk.

1) What's the bottom line? MLB Networks currently cares for 650,000 hours of unique content. The total is expected to consume 50 PB of storage — mostly on tape but some on SANs (see below) — by the time of the MLB All-Star Game scheduled for July 12.

2) Where does it come from? The size of the archive grows every time an umpire yells "Play ball!" — and not just by the length of a game. Consider that each game is represented by a "clean feed" without graphics and a "dirty feed" with stats, a scoreboard, and more superimposed on screen. There may also be multiple dedicated feeds for ISO camera positions, including one or more dugout cameras. The standard minimum for a single game is seven feeds, but Butler said there could be as many as 10 or 11.

3) How fast is it growing? In 2010, Butler said, DIAMOND was managing some 2,000 hours of new footage a week. Does that sound like a lot? Well, over the next six years that number tripled — today, MLB Network brings in some 6,800 hours of video every week, Butler said.

Hours of Footage Added to MLB Network Storage Every Week

Source: Tab Butler

4) How is it edited? MLB Network was forced to reconsider its editorial workflow in 2012, as Final Cut Pro 7 and Grass Valley's Aurora came to their end of life. "We looked at Avid and other platforms, but we needed it to couple very closely with DIAMOND," Butler said. Adobe Premiere Pro, with its panel integration capabilties, turned out to do the trick. As a result, editors can search for footage right inside Premiere, then drag the matching clips right onto the timeline. DIAMOND DASH (that's short for DIAMOND Asset Sequence Handler) works with the AP ENPS newsroom platform and Grass Valley's Stratus Rundown to identify the materials that are being edited and send the finished cuts directly to a playout server. "We need to know about everything on every timeline and in every sequence," Butler said.

5) What's the best interface for logging footage? MLB Network's 20 DIAMOND logging workstations had originally been designed using big buttons on touch-screens for tagging clips with metadata. Users would tap a "pitch" button to start each play, then touch buttons that corresponded to terms that described what happened during the play. However, experiments eventually confirmed that those systems were too slow. Today, MLB Network logs video with a mouse-and-keyboard system that is faster than the original touch interface.

6) How much storage is online? MLB Network has two 2.88 PB SANS, one for the National League and one for the American League. Each SAN runs on the Quantum StorNext 5.2.2 file system and can hold about 90,000 hours of footage  A special "virtual file display" has been created to display the files to users in a way that's easy to understand. "It's the only way to manage the size and scope" of the DIAMOND storage system, Butler said.

7) What moved offsite? The company designed its data center in 2012 and 2013 and had it up and running in 2014. Recording, online storage, editing workstations and DIAMOND infrastructure were moved to 28 racks of VMWare infrastructure at Coresite's nearby location. The data center is connected to MLB Network via two redundant fiber paths.

8) What stayed in the office? The studio playback system, the tape archive, and the editorial personnel all stayed at MLB Network HQ nearby. Editors have access to 84 Adobe Premiere Pro editing workstations running on Cisco UCS C240 2RU rack servers at the data center connected to MLB Network over a single dark fiber path. Each one drives two displays, a keyboard and a mouse over MultiDyne KVM hardware with "no visible delays," Butler said.