Panelists Discuss the Technique and Technology That Will Drive Immersive Content in the Coming Decades
Although virtual reality is barely in evidence in the consumer marketplace, SMPTE deemed it important enough to devote the entire first day of its annual conference, held in Hollywood last week, to it.
As an assembly of film and television engineers, SMPTE is known for drilling down into technology, but that’s not all they did when examining virtual reality. One of the more interesting conversations posed the question: is VR a new art form? The panel discussion, moderated by Blackstar Engineering Chief Executive Andrew G. Setos (formerly Fox president of engineering), included game company Playful Corporation founder Paul Bettner; HBO Creative Lead for Digital Products Colin Foran; VFX artist Vicki Lau, Jaunt Studios Chief Executive Cliff Plumer (formerly CEO of Digital Domain and CTO of Lucasfilm); International Cinematographer Guild president Steven Poster, ASC; and 20th Century Fox Futurist Ted Schilowitz.
The discussion centered on whether VR is an evolution of an existing art form or an entirely new category. “Higher resolution, HDR, [and] HFR are all enhancements but don’t move the experience into a totally new place,” said Setos. “But when people cried at Snow White and the Seven Dwarfs, it was the first time they did so at a cartoon, and people called it the birth of a new art form. Is VR a new art form?”
Foran said he doesn’t think it is a new art form but rather “a logical conclusion for something we’ve been trying to do for a long time.” Lau disagreed. “If you try to see VR as a bigger place to explore the 360-degree environment, there is a reason why we’re still fumbling around and feel that something is missing,” she says. “That could be a sign that it is something new and we have to explore it as a type of evolved entertainment of which VR is the first step.”
Plumer reminded attendees that, so far, technology companies and their demos have driven most of the VR content. “There has to be a creative side to help drive the technology,” he says. “That’s what will build an audience."
Poster brought up the topic of storytelling. “As a perceived experience, VR feels experiential and not narrative,” he said. “There’s a lot we don’t know about how VR is impacting the audience. We’re not reproducing reality. We’re telling a story and getting an audience to suspend disbelief.”
Schilowitz agreed. “We have our dancing shoes on at Fox,” he said. “We’re emerging out of the theoretical and putting these things into practice.” In addition to a VR demonstration based on the film Wild, Fox Innovation Lab with 20th Century Fox is about to launch a commercial VR experience based on The Martian.
A frame from the Wild VR experience created by Felix & Paul Studios for Fox Innovation Lab.
Shilowitz said viewers reported recalling the Wild VR experience as if it were their own memories. “You will believe you were there, that it happened to you,” he said. VR has the power to evoke empathy, several panelists said, and could also be used for documentaries and news coverage.
AMD Chief Architect of VR and Advanced Rendering Layla Mah addressed the technology that will be required to create this new storytelling medium. “We have quite a ways to go … and Moore’s Law might not take us there,” she said, reminding the audience that the first iPod only came out in 2001. “From that iPod to the current iPhone 6 is a big change,” she said. “And VR hasn’t reached the level of iPod 1. VR is at least a decade of job security for semiconductor fabs, GPU designers, 3D modelers and network operators.”
She got specific: “We’re at 90 fps, 1K per eye, 20ms latency, 4 teraFLOPS and tethered,” she said. “To get to reality, we need 120+ fps, 24K per eye, zero latency, 1 petaFLOP and untethered. We want to advance the visual and audio quality and go towards photorealistic rendering, and we’re looking at 15 years and 250x to get there.” If we add gestural feedback, touch and other feedback, says Mah, we’re looking at 25 years and 5,000x to 10,000x. “If you want VR to come to cinema,” she added. “It’s 5 million x — and that’s conservative.”
The answer, she says, is not Moore’s law. “It’s not a straight line of brute force,” she said. “If it is, we won’t see the end of the line in our lifetime. But there are all kinds of other paths we can take. Different architectures can evolve in different ways. The important thing is to find ways to improve quicker than Moore’s law. We need new hardware and software. Audio and visual should traverse the same world together. You want unified software running on unified hardware that traverses the same data structure, sharing synergies. We have to figure out a way to compute reality as one system, as opposed to several pieces. If we do that, I think we can beat Moore’s law.”
Other SMPTE panels focused on the nuts and bolts of creating a VR or augmented reality (AR) experience. Rick Johnson, co-founder of CastAR, a VR/AR technology company, notes that while most people have some idea about VR, AR is an unfamiliar term. “AR is not Google Glass,” he said, noting that some people call AR "mixed" reality. “It is blending a 3D computer-generated scene with the physical world. You are always grounded in the real world. You can interact with people and with the CG objects.”
Unity Technologies’ Pete Moss, who comes from the gaming world, talked about design considerations and production workflow. “VR and AR are very similar,” he said. “Both have some sort of display mounted to the user, both estimate the pose of the user, both have rotational and/or positional tracking, and both work at real world scale.” He said frame rate is the most important factor in producing VR. “Be gentle,” he said. “Don’t drop frames.” He also encouraged VR producers to add even small improvements to the workflow as they become available. “Each optimization has a benefit, no matter how marginal. Optimize every step of the way, as you go.”
Samsung's Project Beyond 16-camera stereo VR rig.
Milk VR senior staff engineer Jeff Wilkinson and Samsung Electronics Tech Lead/Senior Director of Research Sajid Sadi talked about developing VR as background for building AR. Sajid showed all the camera prototypes that Samsung has developed over the years, finally unveiling Samsung Think Tank Team’s Project Beyond, a VR rig with 16 full HD cameras to capture stereoscopic 3D.
Wilkinson dispensed some production tips. “Don’t move anyone in VR in any way they’re not moving in the real world, or you’ll cause motion sickness,” he said. “Color-correction and exposure are more difficult, and there are very few standards. 360-degree videos are the Wild West.”
Milk VR has produced celebrity house tours, professional sports, movie trailers, nature videos, concerts, horror movies and fashion — but Wilkinson noted that Oculus CTO John Carmack once Tweeted, “One of the keys to consumer VR success will be stereoscopic, panoramic cat videos.” Joking aside, Wilkinson said, “VR drives more empathy.” He suggested that VR is good for news and documentaries. “It’s much more personal.”
