Getting Over Technology Hurdles, Promoting Interactivity, and Making a New Form of Journalism

"It's really hard to look cool wearing a HMD," Mark Mine said at SIGGRAPH this week. Addressing an overflow crowd at "Virtual Reality: Creating at the Edge," a Tuesday-afternoon talk, he was summarizing some of the issues facing VR today, and that meant taking a critical look at the ubiquitous head-mounted display (HMD). He was (probably) half-joking about the HMD, but the comment was apt. With only first-generation products ready for the market, everyone doing VR at SIGGRAPH is trying to look cool — but there's still a lot of stumbling around going on.

Mine is founder and director of the Creative Technology Studio (CTS) at Walt Disney Imagineering Research and Development, and that means he's done a lot of thinking about this stuff. He said the HMDs have the advantage of being relatively cheap to manufacture, offering a reasonably wide field of view. And lately, he said, HMD makers have come to recognize that the human field of view is "biased downward," and are doing a better job of adding degrees of view in that direction, and not just out to the sides–though they don't come anywhere near covering the entire human field of vision.

The main issue with HMDs is that they are an isolating technology, Mine said, which can create cultural problems when people congregate in groups of two or more. Because Disney is focused on group interaction, CTS has been developing the VR experience in caves — small dark rooms with 360-degree immersive imagery projected on the walls — since they are more amenable to group experiences.

Mine described the newest CTS Digital Immersive Showroom, or DISH 2.0, as a cave measuring 10.5m by 7.5m by 4.0m, with images projected on the walls by five 4K 3D projectors running at 120Hz and one 2K 3D projector (for the back wall) also running at 120Hz. The room's design, with rounded corners and an overhanging ceiling, can get rid of shadows in the corners of most caves, he explained.

Mine said many of the issues that remain to be solved for VR are questions of numbers. Generally, they need to be higher. For example, doing justice to 20/20 human vision requires dedicating about 60 pixels per degree in the field of view. For a 120-degree field of view, that comes out to about 8K. "We've got a ways to go to get there," Mine said.

Another physical specification to take into account is dynamic range, Mine said, with the human eye managing a contrast ratio of about 1000:1 instantaneously and 10,000,000:1 over time. New HDR projection technology is achieving about 1,000,000:1, he said. Mine also discussed brightness and color gamut, but warned that new multi-primary displays may introduce their own problems with regard to the way people perceive the color of narrowband light soures.

Frame rates figure in, too, with cave applications coming at 30 to 60 fps and HMD sets generally running at about 90, Mine said. He noted, however, that theme parks are pushing frame rates to 120 and higher. And, of course, displays are required to adequately handle those frame rates with low latency and low persistence.

Other speakers discussed their own experiences with VR technology and creativity. Richard Marks, director of the PlayStation Magic Lab at Sony Computer Entertainment, described Mighty Morphenaut! [pictured, top], a research project conducted in collaboration with NASA's Jet Propulsion Laboratory. The result was a simple interactive game that allowed two players wearing HMDs to face each other's Robonaut 2 avatars while performing simple grasping tasks. The VR images ran at 120Hz using OLED technology and provided a 100-degree field of view.

Sony learned a few things from the experiment, Marks said. For example, being able to hear the other player's voice, especially their laughter, was crucial. "Voice adds a lot to the VR experience, especially when it's directional," Marks said. And while the game animated the robots' hands, it did not animated the robots' heads, so one player could not tell what the other player was looking at. Marks said that was probably a mistake. The research will inform Sony's continuing development of Project Morpheus, its own PlayStation 4-based VR platform.

For all the technological savvy on display, the most out-of-the-box thinking must have been demonstrated by Nonny de la Peña, an experienced journalist who is now CEO of Emblematic Group, which creates immersive applications with the goal of evoking "empathic engagement." She said the precedent for this kind of work dates back to Walter Cronkite and his 1950s television program You Are There.

De la Peña showed a series of examples, beginning with a Gone Gitmo video game designed to teach users about the experiences of prisoners at the Guantanamo Bay prison camp. Players were not actually placed in painful poses — they were merely seated in chairs with their hands behind their back — but the VR scene included a mirror that showed a person hunched over in a stress position. She found that participants felt in retrospect that they remembered assuming the physical position of their avatar in the game.

She developed the same concepts further with Hunger in Los Angeles, which used real audio recordings to restage the scene outside a Los Angeles food bank when a man collapsed and had a seizure while waiting in a long line for food. That project was accepted at the Sundance Film Festival in 2012.

Other VR experiences from De la Peña and her team include Use of Force (see video above, from a Buzzfeed feature on the project), about a man who was beaten to death at the San Ysidro Border Checkpoint, Project Syria, which places players in a Syrian city as a terrorist bomb is detonated, and a project about Trayvon Martin, the hoodie-clad Florida teenager who was shot and killed by George Zimmerman, that was turned around in less than two weeks.

In all cases, the team acts as journalists, collecting as much accurate information as possible about what actually transpired during a given event in order to faithfully recreate the scene for viewers to experience through VR technology. De la Peña calls it "immersive journalism," and she thinks it has the power to engage viewers with well-reported stories in entirely new ways. That's a good thing. If it's hard to look good when you put the goggles on to enter virtual reality, you may as well be learning something important while you're there.