A Q&A with StorageDNA President and CEO tC Chakravarty

Despite many advances in spinning-disk technology, when it comes to archives, tape still makes a lot of sense from the standpoints of both cost and manageability. With the advent of LTFS technology, which makes file systems on tape more disk-like, some of the headaches of managing a media archive on LTO tape have gone away. But tape is still tape, and getting material from linear media as quickly as possible remains a challenge. StorageDNA President and CEO Tridib "tC" Chakravarty says the company's direct access technology makes an LTO archive operate with the efficiency of a nearline disk archive. This week at ExCel London, StorageDNA demonstrated a shared-storage and LTO LTFS archiving workflow in conjunction with Facilis Technology. We asked Chakravarty about the limitations of tape storage, what purchasers need to know when making investment decisions, and what his company's archive software brings to the table. We also asked him what can go wrong, and how users can plan for disaster.

StudioDaily: Most of our readers know that LTFS is a file system that makes LTO tapes behave more like disk-based storage. What else do they need to know about the technology?

tC Chakravarty: LTFS has really changed the way people look at tape as a storage medium, while providing an open, low cost, and reliable alternative to spinning disk. First, it is hardware and operating system agnostic, allowing for total flexibility in today’s complex media environments, with mixed operating systems and various creative toolsets. Next, at less than 5 cents per Gigabyte, LTFS-formatted LTO tape can greatly reduce storage costs for media-centric organizations trying to manage the deluge of content they receive on a daily basis. And finally, the future of LTFS is secure through support from the LTO Consortium with a published roadmap for LTO extending out many generations, providing peace of mind for those looking to create an open and accessible archive of content for years to come.

Is LTFS a mature standard, or is it still evolving?

LTFS was introduced with LTO-5 generation drives and associated tapes. Today’s current shipping version is LTO-6, so in those terms it is a fairly new standard and is still evolving. At StorageDNA, we made a conscious decision early on to exclusively support LTFS with our archive solution, DNA Evolution, because we had confidence in both LTFS and LTO technologies. That being said, we see positive changes and improvements in LTFS drivers on a regular basis, which we are able to quickly take advantage of due to our expertise with the technology.
     
Which features are the most important in making an LTO LTFS archive efficient enough to qualify as "nearline?"

In order to accelerate 4K and other high-res media workflows, it’s key to have seamless direct access to files on LTO which, on the surface, LTFS provides. However, LTFS by itself is not the answer. What’s also necessary is an infrastructure that includes an intelligent software layer that enables third-party applications to interact with LTO, such as APIs, watch folders, and scripts that can provide direct file access. This infrastructure, created by combining direct access technology with the benefits of LTFS, can offer direct file access from LTO to streamline the process of repurposing and monetizing archived content.

When you say nearline, what do you mean? How fast and accessible can an LTO LTFS-formatted tape archive really be?

LTO LTFS drives are actually very fast, with LTO-6 drives capable of 160MB/s performance. But it’s important to note that there are other factors that come into play when measuring the speed of a tape archive. While LTFS makes tape look like a disk at the OS level, a key differentiator between tapes and disks is that tape is a sequential access medium. This means that for LTO and LTFS to become considered nearline storage some extra help is needed. At the SMPTE 2014 Annual Technical Conference in Hollywood, I presented on a direct access technology that allows tape to behave like nearline storage. To summarize, the biggest challenge when presenting media files to applications directly from LTO via LTFS is the way that media applications read media files. Meaning, they rarely read files from beginning to end in a sequential manner. In fact, modern media files are quite sophisticated, with a single file containing numerous bits of metadata, all of which are randomly requested by an application when reading the file. Because files typically reside on disk (a random access medium) delays caused by random requests are not noticeable, but on LTO, these requests often result in long wait times and, ultimately, applications can hang. StorageDNA’s technology intelligently pre-reads the appropriate metadata portions and stores them in memory. With this simple, yet highly effective feature, applications are able to read files directly from LTO via LTFS without the need to restore them, thereby providing nearline storage capabilities from tape. This direct access technology has been proven with LTO LTFS in our lab and our plan for introducing it to the marketplace is in process.

LTFS is an open standard, but what about the archive software and metadata?

This is a great question because this very important factor is often overlooked. There are many archive solutions on the market and most are proprietary in one way or another. Data and metadata must be completely open to guarantee that content will be available and accessible long-term, which is a requirement for any archive solution. StorageDNA’s archive software [DNA Evolution] offers 100% open data thanks to LTFS. Plus, the catalog that contains information and metadata about all files that have been written to tape is in an open XML format, providing 100% openness across the board.

What questions should users ask when shopping for an LTFS-based archive system?

• Can the archive solution provide media-centric workflow benefits? If so, what are these?
• What sort of integration can be done with my existing environment and/or toolsets?
• How scalable is your solution?
• Is the archive solution proprietary or open?
• What operating systems does your solution support?
• What transfer speed performance can be expected?
• How many deployments with similar workflows to mine do you have? Can I contact some accounts?

What new workflows are you seeing being adopted for LTO with LTFS?

Due to LTFS’ inherent openness, we see StorageDNA customers taking advantage of new workflows with DNA Evolution that help to streamline media pipelines. One example is a camera master archive workflow, where LTO with LTFS tapes are used as a footage repository. The need for this workflow is driven by the high volume of acquired content and the requirement for a safe and reliable place to put it without having it overload work-in-progress spinning disk volumes (local, NAS or SAN). Another benefit is this allows for the quick repurposing of very expensive memory-based acquisition storage (such as, P2, XDCAM, and REDMAG). Another unique workflow that DNA Evolution users are taking advantage of is the ability to do an “old school” style of offline-toonline conform, directly from LTO. This automated conform workflow eliminates long manual conform processes and the time-savings helps lower post budgets.

What are the biggest limitations of LTO with LTFS technology?

The inability for LTO tape drives to read back beyond two generations is probably the biggest drawback of the technology. That being said, it is always a good idea to plan properly for the migration of data to newer technology generations over time, regardless of whether you use tape or disk for your archived content. 

What’s the biggest potential disaster that could befall an LTO tape archive, and how can users prepare for that and minimize any risks?

Not planning for disaster is a crucial mistake. If there is content in an archive that is known to be irreplaceable, then multiple copies of this content must exist–ideally in different locations. LTO with LTFS as a storage medium makes this easy because of its size and portability. Couple that with duplication features, as in our DNA Evolution software, that allow for easy duplication of tapes, and you have the ability to minimize disasters involving loss of “one of a kind” content.   

What are the biggest mistakes people in production and post make when they think about storage and archiving?

The biggest mistake is not having a plan upfront that includes not only work-in-progress spinning disk storage allocation, but also an archive/backup component. Most people don’t think about archiving until capacity constraints begin to impact their ability to get their work done, and ultimately, get paid for jobs. Many others push it under the carpet until it becomes a major issue, such as a catastrophic loss of invaluable content. StorageDNA helps customers transition their content from spinning disk to tape and also helps them move content from racks of USB hard drives over to a much more reliable and accessible solution.

tC Chakravarty is a seasoned executive in the storage and archiving industry. As StorageDNA’s founder, CEO and president, tC provides the vision and strategy that drive the business. Prior to StorageDNA, he was with Quantum’s Advanced Technology Group, where he helped map key strategic initiatives and contributed to the development of its de-duplication and continuous data protection (CDP) technology. Earlier, tC was part of the initial team at Panasas that developed PanFS, one of the world’s most advanced clustered file systems. tC earned his BS and MS degrees in computer science from Carnegie Mellon University.