Mid-way through the animated mockumentary Surf’s Up, one of the penguins faces the camera and asks an invisible camera crew if they’re hungry. We hear a voice answer, “Uh, yeah.”

CG features often mimic live action cinematography. Even though the virtual camera has no limit, scenes filmed with real world limits help make the 3D world seem familiar and therefore believable to movie-going audiences. But in creating its latest feature animation, Surf’s Up, Sony Pictures Imageworks has pushed that notion to an extreme. The movie we see is the documentary being filmed.

“It’s what attracted me to Surf’s Up,” says co-director Chris Buck. “I’d never seen a full length animated documentary. We could have a character do an action and be interviewed about it. That’s unique.”
For the animated features The Polar Express and Monster House, Imageworks had invented a system dubbed “Wheels.” A camera operator filmed shots of CG characters animated with motion-capture data using a standard camera system altered to give the operator a view of the virtual characters.

To give its hand-animated film a cinema verità© style, Sony once again devised a system based on a real camera. But this time, they created a handheld camera.

The Universal Pictures feature directed by Buck and Ash Brannon and based on a story idea by producer Chris Jenkins follows a teenaged surfer, Cody Maverick, from his home in Shiverpool, Antarctica, to complete in the Big Z Memorial Surf-Off. In Shiverpool, the documentary crew interviews his mother, chopping fish inside the family igloo, and his brother, a traditional egg sitter. But, most of the film follows the action on tropical Pen Gu Island where Cody meets champion surfer penguin Tank the Shredder, falls in love with Lani, the beautiful lifeguard penguin, hangs out with Chicken Joe, the film’s only surfing chicken, and learns life’s lessons from the Geek.

“We boarded the film like any other animation, but the trick after that was to create the illusion of spontaneity,” says Brannon. “That you aren’t quite sure what a character was going to do at any moment and you had one chance to catch it with a handheld camera.”

But capturing that documentary feeling with a virtual camera is difficult. “The movement of your [the camera operator’s] body position, the rise and fall of your breathing, the subtle nuances of motion,” says James Williams, layout supervisor, “all contribute to the look of a handheld camera.”

By using a real camera, the layout artists, who act as a camera crew on animated features, wouldn’t need to try to fake that motion. At first, Williams expected to use the system only for sweeping shots that would be especially difficult to give a handheld look in the computer. But, the realism achieved by physically placing a camera on his shoulder became addictive. “It was the first time I felt as a cameraman that I truly knew where I was and thus could place the audience in that position, too,” Williams says.

The entire process ran through Autodesk’s Maya. What the camera operator saw in the viewfinder of the handheld camera was the animated scene as seen through a Maya camera, thanks to development work early on by the Imageworks crew. “We stripped the geometry to the essentials and optimized it to a great degree,” says Williams.
As a result, when the camera operator turned, the scene in the viewfinder moved as a real scene would in the real world. And, because this all happened through Maya, there was no need for data translation. The animation team got exactly what a layout artist shot. Quickly.

“We created the system to get a handheld feel,” says Rob Bredow, visual effects supervisor, “but it was much faster than having to keyframe the camera moves by hand. We could prep a file, shoot it to layout, and they could do 20 takes in 20 minutes.”

Williams estimates that the layout artists filmed between 85 and 90 percent of the movie with the handheld camera. “We never dreamed it would be so successful that we’d use it on nearly every shot,” says Williams.
The camera was a Sony M3A bought for $250 on eBay. “It has an eyepiece with a small black-and-white CRT screen on the camera,” says Williams. “We attached that to our monitor output so we could look through the eyepiece and see the shot.”

The shot they saw was of characters at an intermediate stage of animation. “Usually, you start animation with a rough camera layout, then you animate,” says Chris Juen, digital producer. “In final layout, you limit the camera movement because it affects the approved animation. But we didn’t want to limit our camera.”

Thus, for Surf’s Up, animators blocked out the characters’ actions, layout artists filmed the characters, and then animators polished the animation and added the facial performances to have the characters react to the camera.

“The characters are self-aware,” says Jenkins. “They might give an answer, but not tell you everything. For example, Cody is asked a lot of questions on camera and sometimes the camera stays on him just a little bit after he’s finished speaking. That additional second of information is more telltale than what he just told you. The animators were quite brilliant. Sometimes his eyes say more than the vocal performance.”

To film the action, layout artists created a 12-by-12-foot “stage” by moving their desks against the walls of their room and taping markers on the ground. “Animation has no constraints,” says Williams, “but most movies made on location are constrained by the size of rooms they’re in. We didn’t go so far as having the camera crew bump into a wall, but we did stay close to the reality of an environment.”

On the ceiling above the stage, the crew attached an array of more than 3,000 infrared LEDs arranged in a grid. A lightweight, HiBall optical sensor from 3rd Tech attached to the handheld camera, viewed the array with its six lenses and rapidly calculated the camera’s position based on feedback from the array. “The beauty of this system is that it’s incredibly accurate because it’s attached to the object that moves,” says Williams. “A small rotation on the camera translates to a large amount of offset on the array itself.”

To add authenticity, the crew shot with wide-angle and telephoto lenses. They could even zoom using a rocker switch close to the lens. “One of our engineers created a system so we could encode the data from the camera and drive the virtual zoom in the lens,” says Bredow. “It’s velocity sensitive. If you push harder, it zooms faster. We wanted the camera operator’s experience to be intuitive.”

It was so intuitive that when the character Geek in the film threw shells at the camera, the camera operator in the layout artists’ room ducked. Similarly, the camera quickly moves out of the way when Cody accidentally swings the surfboard toward the camera crew. Often, the crew simulated the actions a real camera crew might do ‘ on purpose, or not. They ran. They fell down. “When one of our characters falls over a cliff, the cameraman looks around to see if anyone can help,” says Williams. “And in a sequence when Cody is injured, the camera crew desperately tries to get footage of Cody being treated.”

In addition to the handheld camera, the crew also devised other types of cameras. “We start with Cody in Shiverpool, and imagine that we have a small crew,” says Williams. “One cameraman who could go into the water, and another shooting the documentary. As the film progresses, the number of cameras increases.”

A TV crew is on the island, for example, shooting the competition with helicopter cameras, and the documentary crew uses their footage as well. “We translated the camera in Maya using motion paths to describe the motion of a virtual helicopter in our environment,” Williams says, “and then applied rotations from our motion-capture camera to simulate the feeling of a camera crew looking out the window.”

Similarly, to simulate the look of a camera filming from the back of a jet ski, the crew translated a camera in Maya and applied camera shake from the real, motion-capture camera.

Also, Imageworks built camera rigs in the water that could displace themselves with the movement of water to replicate the motion of a camera fastened to a surfboard shooting through the waves.

Imageworks modeled the waves using blend shapes, which allowed animators and layout artists to treat them as characters. “We animated the wave profiles in a similar way to the way we animate face shapes,” says Bredow. Previz tools gave the layout artists and animators a way to see the whitewater and a surfboard’s wake.
“The wake gave the animators the speed and direction,” says Bredow. “It helped them know how to point the surfboard.” Animators could use a “wave rider” to attach a surfboard to the wave, or hand-animate the board on the moving background, and place characters on the boards using offset nodes with moveable pivots.

The previs tools also helped the layout artists. “The cinematographers in surfing movies compose for the wakes,” says Bredow. “And the wake enabled our camera operators to frame the shot properly.”

One of the consequences of using the handheld camera is that many shots in this film are long, as they would be in a documentary. “We have probably 300 to 400 fewer shots than a typical animated feature,” says Williams. “This was one of the most exciting and interesting movies I’ve ever worked on from the point of view of pushing the ability of animation to simulate live action.”

There’s a good chance he and other members of the Imageworks team will have future opportunities to use the new camera system. “This tool has been so well received that it seems it’s clearly the way to go any time we want a handheld camera,” says Bredow. “It terms of the audience ‘ it’s easier to laugh with a handheld camera. I think we’ll have lots of opportunities to use it in animated films.”