Letteri is now the director of Weta Digital and senior visual effects supervisor. In 2011, he led the artists at that visual effects company who created The Adventures of Tintin, which won a Golden Globe and received a BAFTA nomination for visual effects, and he led the visual effects work on Rise of the Planet of the Apes, for which he received his sixth BAFTA and sixth Oscar nomination. He shares the Oscar nomination for Planet of the Apes with visual effects supervisors Dan Lemmon and R. Christopher White, and animation supervisor Daniel Barrett all of Weta Digital. Rise of the Planet of the Apes stars a chimpanzee named Caesar, a CG character performed by Andy Serkis and created at Weta Digital, who acts alongside actor James Franco, a scientist who raises Caesar like a child. Weta Digital created all the visual effects in the film.
Joe Letteri: From a physical point of view, all the same cues that you see when you photograph a real chimp are there. The shape of the body, the way the hair looks, the way the skin scatters light, the way the eyes reflect light. Then, from a performance standpoint, you have the subtlety of motion that you can get when you capture the performance from an actor. You get a sense of overlapping motion and intent that an actor brings to a scene. That physical performance is complex. It’s possible to do it by hand, but it’s really hard. You think of keyframe animation as pose to pose, but in human life, we never go from one pose to another. Every part of the body is in motion. When you capture motion from an actor, you get that natural fluidity, that overlap. So you have that instant recognition of physical movement that you’re familiar with.
But Andy Serkis isn’t a chimp.
No, but many of the ways a chimp moves are still familiar to us as humans. You can look at a chimpanzee arm and a human arm and see the similarities, and the way in which they move is similar. There are differences. Chimps are on all fours and that’s hard for human actors to mimic, but they did. They trained for that. They had arm extensions. They learned how to walk on all fours. Then, we distributed the weight to compensate for the chimpanzee’s movements.
Is that why you used performance capture?
We’ve been doing character work since Gollum. For that film, we hand-animated the face, but for a lot of scenes we had Andy [Serkis] re-do the [live-action] performance on a motion-capture stage. We continued improving our performance capture until Avatar, where we were able to capture the face and body together. With Apes, we took that idea further and got performances on the day, which brought it full circle. My preference is to work with actors as much as possible. You get the realism we were talking about, and the directors and actors work out the scenes. It bridges the gap between live-action filmmaking and animation. Look at Andy’s [Serkis] performance – that was a great performance, but because you can’t see his face he isn’t recognized for it. People aren’t sure where to put these things. Is that animation or is that Andy? It’s both.
How were you able to capture the performances outside?
We used an LED suit because that let us work on stage. It didn’t interfere with the camera and we didn’t have to pump a lot of light into the scene. Only our motion-capture cameras could see [the LEDs]. It worked in bright sunlight even with all the reflections coming off cars. We redesigned our technology to create a good, clean, portable system.
Did you do performance capture for other apes, as well?
We had five featured apes, three versions of Caesar, and 15 or 16 background apes that we replicated for crowd scenes. We had actors assigned to each ape. Sometimes they would swap around. For stunt work, we would have stunt actors for Caesar, just like in live action films. Sometimes we’d keyframe the stunts if the characters had motion the actors couldn’t do. Because we had the actors on set, we were able to capture the interaction with actors playing the humans in the film. Then we painted out the actors and painted back any missing bits. It was a lot of work compensating for the size difference of Andy and a normal chimp. If an actor touched a chimp, sometimes we modeled a virtual hand to interact with the digital chimp.
How did you make Caesar – and the other apes – look so real?
The more we’ve studied hair, skin, eyes over the years, the more we’ve realized that it almost comes down to a brute force approach. You have to calculate everything to get it right. Millions of hairs, subsurface scattering, all the detail in the eyes. We develop technology to allow us to include the maximum amount of detail in the shot, rather than to find the amount of detail we can “get away with.”
You start with the model of a chimp and have that basic form, but a lot of what you see are muscles that you don’t put in the model because they move. You have to build the model, take out the influence of the muscles, put them back in, and try to get the shape you want to see. You try to see how muscles move in a real chimp.
Did you develop new tools for this show?
In the old days when we did muscles, they were influence objects. First, there was skinning, range of motion, proximity of skin to skeleton, and then influence objects like balloons under the skin. We’ve now abandoned that completely and gone for a full physical model. We have a full skeleton with all the muscles and fibers modeled. We use finite element analysis, which is a basic technique for calculating stress and strain. We have a model for how the fat layer combines with the muscles. We talk about how the skin slides over muscles, the combination of skin and muscles moving together, counter to each other. So, yes, we came up with new tools.
Also, we had new tools for hair. On King Kong, we could groom hair procedurally – have this area go left, this one go right with a wave. We abandoned all that. On Apes, our groomers could deal with every hair directly. They can control every hair down to a single hair. At the end, this gives you more control. It’s more like the real world. They could get in there with a comb and scissors and start grooming.
It sounds like you’re trying to lose all approximations.
Yes. I found that we spent more time trying to figure out if we had it right rather than getting it right.
Is that true for lighting, as well?
We haven’t gotten to the point of using real lights; you don’t need to mimic the limitations of physical lights. What you need to understand is the behavior of the light transport. You’re not restricted by the design, you’re restricted by the physical response. If the [physical] light tubes are 1 meter, there’s nothing wrong with making [the digital tubes] 1.5 meters as long as we calculate the light transport properly.
Did you also create digital environments for the film?
Two of the big scenes were inside the primate center. We photographed there even when we had performers. But when the apes are doing stuff at the top of the ceiling, those shots were all digital.
Then there was the Golden Gate Bridge. We had two establishing shots, one early on and one in the third act that were aerial photography. The rest were digital extensions of a section of the road that we built, 100 meters long. We dressed it to look like the deck of the Golden Gate Bridge and dropped that into our digital bridge. The whole Bay Area and all the water are digital.
Muir Woods was a combination of a park in Vancouver and digital sets. Ground level was all shot in the park, and then once you see Caesar climbing into the tree, it transitions to digital. We used the idea of seeing San Francisco from [nearby] Mt. Tamalpais, but you know, you can’t really see San Francisco from Muir Woods.
Did you have a favorite shot in the film?
The best thing was when we could really read Caesar’s personality, like in the first shot of him in the cell watching after he’d taken the intelligence-enhancing drug. You can tell he’s thinking about what’s going on. You can see the intensity of thought. The feeling that this is a thinking animal was strong. Those moments wake us up to the possibilities of a character like that.
The hardest shot?
A couple were really difficult. That particular one with Caesar was hard because it was the first one and it was subtle. The scene where he’s a toddler, when he grabs a cookie and goes upstairs, was hard because we had to stitch it together from a lot of plates and had to keep the animation alive.
The ending shot was tricky – the choreography, and then the reveal at the end. It was on a compressed schedule, and it changed at the end, which added to the complexity. Those are the ones that stand out. But the shots on the bridge were complex because there was so much action and so much to build digitally.
And, in general, the fact that Caesar had a track suit on top of his fur didn’t make life easier, either.
What trends do you see in visual effects?
We start with the script to determine what is needed and work from there, but we build techniques with an eye toward the future. We think about the fundamentals of what makes these things look real. If we can use those ideas to create creatures, environments, buildings, something we can relate to, we’re just that much ahead.
This is your sixth Oscar nomination. Do you remember the first?
It was exciting. When you’re working on this stuff, you don’t know. All your focus is on getting shots done and getting the movie finished in time.
Did you enjoy this article? Sign up to receive the StudioDaily Fix eletter containing the latest stories, including news, videos, interviews, reviews and more.
