Getting the Shot, Putting It Together on Set and in Post, and Immersing the Viewer in 360-Degree Storytelling

Cinematographer Andrew Shulkind is an avid experimenter. From film to digital, and from cameras to lenses to lights, he revels in putting new technology through its paces while still keeping busy with film and commercial work. That driving curiosity is likely the reason he’s an early adopter of virtual reality technology, spending time out in the field to see what works, what doesn’t work, and what constitutes the new visual grammar of 360-degree cinema. It's culminated in the creation of Headcase VR, a filmmaking group describing itself as a "360-degree virtual reality content studio." We asked him about shooting a truly immersive environment, the special challenges of lighting a 360-degree scene, and the workflow pieces, including multi-headed camera rigs, that make virtual reality come together.

First let’s talk about terminology. We refer to “360-degree” images, but isn’t it possible to capture a VR environment that doesn’t include all 360 degrees of view?

Virtual reality is all about virtually transporting the viewer to convince them that they are somewhere that they are not. A full panorama is central to that illusion. The viewer must have the freedom to look in any direction. 360 degrees longitudinally by 360 degrees latitudinally offers the most complete immersive experience. We did some early tests shooting only 270 degrees so that I could hide my Steadicam operator, and to cut down on the number of cameras required, but it just wasn’t a full enough experience. So in terms of building camera rigs, i’ve been focused on having the capability to shoot 360 degrees by 360 degrees, and we pull cameras when they are not needed. And when mounted on a drone or in some other rigging circumstances with a perspective of only a plain blue sky or green grass, we just paint in that angle. The camera always needs a mounting point, so we put it in the most unassuming place possible, depending on the application—above you, below you, or at a rear 45-degree angle, meaning you can look up and down but not back through your body.

So how did you get involved in VR?

I’m always looking to enhance stories in nuanced ways—and I am an optimist about technology, so I get intrigued when something sensible comes along. And each project has its own unique requirements, so I’m always looking for unique ways to fulfill those. Over the past 10 years, besides shooting using Alexa, Red and Canon for my A camera, I’ve been using tiny cameras like the SI-2K, Iconix and Cunima cameras for specialty purposes, including shooting stereo 3D, and recording to external devices and moving the camera in different ways. Taken together, all of these technologies created a perfect composite for the way of thinking and tools required for shooting VR.

When some producer friends asked me to help with building out the capture aspect of a new VR company focused on cinematic storytelling, I couldn’t say no. It’s been a slippery slope since then. We’ve joined several VR partnerships with companies including Codex and a few others that I can’t mention yet.

Your company is called Headcase, and pictures of the Headcase Cinema Camera have turned some heads. What, and who, exactly, is Headcase?

Headcase is a partnership of filmmakers from the feature, advertising, and sports worlds. We produce professional complete 360-degree virtual reality solutions in live action with the same efficiency and style that we use to shoot movies and commercials. One of our partners is producer Lucas Foster [Mr. and Mrs. Smith, Crimson Tide, Man on Fire], who has really led the storytelling charge, challenging us to find ways to drive a narrative forward while still giving the viewer a sense of autonomy. The focus of the company is telling these compelling cinematic stories with the same high bar for filmmaking that we observe in our other arenas of expertise. And since the transportative aspect of storytelling is common to all cinema, we reflect those requirements back into our 360-degree VR work. 

This space is so new and developing so quickly that we’ve come across a lot of equipment, solutions or attitudes that didn’t make sense to us. In those cases, we’ve just built our own version of worked best for us. This was the case with our camera systems—I wasn’t satisfied with what was available, so I built my own. Now we are on version 3.0 of our high-resolution camera system and a new build of our smaller Headcase Cinema Light rig. Also Headcase created a customized pipeline from capture to stitch, to editorial to delivery. If you go down this road, every aspect of this process has to be reimagined. We wanted a modular system that could interface cleanly with our film- and commercial-friendly way of working. Every step of the Headcase process was built imagining a client sitting there, or a producer looking over your shoulder. It’s well planned and fully functional. Also, the VR arena offers a lot of opportunities besides gaming and entertainment, and we are already servicing some of those. 

I understand the camera is available through Radiant Images?

That is correct. I have had a long and trusted relationship with Radiant and they are one of the first places that I thought of when it came to testing, proving and building something like this. They are one of our official partners, and Michael [Mansouri] and Sinclair [Fleming] have been instrumental in helping me fabricate and develop this technology. Radiant oversees the rentals and services our equipment.

Can anyone rent it?

Yes, they can. The rental situation is a bit unique. Headcase is fully turnkey, so the idea is that we can cover as much of the process as you like — from partnering on creative and producing the job from capture to VR deliverables and bulk-branded VR goggles or cardboard — to only delivering stitched raw files, and everything in between. Each project is unique, but we do end up collaborating a lot creatively because our experience in the field means we know what works and what rules can be rewritten. 

Does it always involve you as the DP?

Of course—I have to have some job security! But seriously, I have a great team of DPs, operators, and assistants. If I can’t personally do the job, we have a team that has worked with the rig and knows it well. Our rigs are easy to use, but they are delicate babies.

You mentioned Codex. What’s their involvement?

I’ve been using their equipment for years. Their Codex Action Cam is an essential building block in our system. I am working with them on a couple of projects, and they are poised to dominate the VR space. 

What does all this VR stuff mean for you as a DP? Are you quitting your day job?

Not at all. Shooting traditional commercials and movies is still the majority of my schedule and I’m still loving the range of possibilities and opportunities there. VR is another tool in the arsenal and an additional way to tell new stories in new ways. The exciting thing about this fully immersive space is that it dictates a completely new kind of filmmaking vernacular.

What do you mean?

­­

It’s been an interesting process of discovery. We have done several rounds of extensive testing and figured out surprising new ways of covering scenes. In general, the more autonomy the user feels in terms of viewing field, discovery and movement, the more immersive the experience is. But more autonomy also means less narrative drive, and the narrative is what maintains interest and sells the product. Without the narrative, it gets boring quickly. So it’s a delicate balance.

In one of our tests, we explored how to cover a dialogue scene in 360-degree VR between two people. We found that an over-the-shoulder feels creepily voyeuristic, and a wide shot feels alarmingly distant. By far, the most immersive and natural way to shoot was to put the camera right in between the characters so that the viewer could choose who to watch. It was an unexpected discovery.

What are the other challenges of shooting in VR?  It sounds like it requires a big footprint.

The lighting constraints are the hardest for me, because I really love using light to focus the viewer’s eye. You have to continually remind yourself in VR that you see everything. My lighting has to become much more global, or come from outside an obscured window, or just [be] hidden. In the best-case scenario, the upward angle gets painted in and I can light how we need to from above.

Monitoring is an issue. Every single system out there has everyone looking at a patchwork of multiple video streams individually or on quad-split monitors. So we are finalizing our unique semi-stitch, which is an onboard live stitch for viewing. It’s a huge and critical innovation. But it’s only for monitoring. We still stitch the 12-bit raw files for the master and deliverables.

So there are challenges to be sure, but really no more so than on any production, just reallocated. Shooting is very similar to a normal job. It becomes easier with fewer crew but requires more explicit blocking.  Post is still whatever it would be on a 2D job, just multiplied by the field of view. We add a stitching step, and also post convergence.

Moving the camera is actually simpler because there are only a few ways to do it that don’t induce dysphoria. So we have a pretty slick platform to do that. But a fixed camera position is extremely effective for the right application. So it requires us to get creative and think broadly about how to pull off the job. Again, it’s pretty similar to all the other stuff that I do—just different tools for the job. 

Speaking of convergence, what about 3D?

In spite of my experience shooting stereo 3D, I have found that in VR it makes a lot more sense to post-converge for 3D. In the traditional filmmaking world, I always shoot 3D with parallel convergence and dial in the degree of depth afterwards, but the distance to objects in VR is so critical that shooting stereo pairs doesn’t allow you to get the cameras close enough together to shoot closer than 15 feet away. Our current VR rigs have between four and 17 cameras, so it would double that number. We have partnered with a well-known provider of 3D post convergence from the movie world for all of our convergence. They are extremely fast, and the process helps streamline the stitch too.

There are exceptions to this rule. We’re building a rig that will shoot stereo pairs for an upcoming project because it will have a live stitch and there won’t be the time for the post work. But that is a largely aerial job, and close distances aren’t a problem in that case. It really depends on the project.

Also, it is worth noting that Youtube’s native 360-degree capability—and possibly also Facebook’s, though it’s unclear at this time—will not be in 3D. You will be looking at a 2D window, so the convergence step and 3D origination aren’t even necessary.

Here’s the big question: how expensive is it?

Less expensive than you might think. It isn’t a low-budget medium, but I’ve collaborated on several rigs in partnership with other companies now, and I can tell you there are high-end and low-end rigs. For me, the baseline is high-resolution, non-GoPro images that we can capture reliably for a client. And that can be handed off to our post department for an easy, fast stitch.

What do you have against GoPro?

[Laughs.] No problem at all. I own several of them! And I’ve used them on projects big and small. But they are fiddly to work with on a professional set, especially in the VR space when you have a bunch of them that have to work together. If one camera overheats and goes down, you lose your 360-degree view. You can’t genlock them, so getting a reliable image from a bunch of GoPros is not possible. Also, the lenses are low quality, and those tiny chips are hard to manage. We have hacked them for a cheaper rig and they have some incredible features, but it’s not prime time-ready when you’re racing the clock and the thing has to run or else you miss the shot.

Is there a case to be made that VR or “immersive” content is easier to monetize?

I think there is a great case to be made to advertisers and companies with something to sell, which is where the money is right now in this arena, that immersive experiences are the ultimate form of targeted marketing.  Broadcast commercials, radio spots, print ads, online pop ups, even integrated branding are all something that a viewer can turn away from. With 360-degree VR, you are marinating the viewer in your message.

There’s an image! What does the future look like? Where do you see this business going? Can you imagine it replacing the theatrical experience?

Younger audiences are consuming content on smaller screens, so I fear the end of the theater experience in the next several decades is a foregone conclusion having nothing to do with VR. The bottom line is that VR is here to stay in some form or another. I don’t think all content should be VR, just like I don’t think that all content should be 3D. I don’t even think that all VR content should be 3D. But someday in the future, a new generation of humans will think it is quaint that we ever watched content shaped like a rectangle.