Digital Doubles, CG Water, and the Difference Between Creatures and Characters

Joe Letteri, Weta Digital’s senior visual effects supervisor and studio director, extends his long list of awards with his eighth Oscar nomination, this for supervising the visual effects in Peter Jackson’s The Hobbit: The Desolation of Smaug, an MGM production released by Warner Bros.

Letteri began his career at Industrial Light & Magic, where he was a CG artist on Jurassic Park, became associate VFX supervisor on Mission: Impossible, and moved up to VFX supervisor on Magnolia. He joined Peter Jackson’s Weta Digital to supervise the visual effects for the 2002 film The Lord of the Rings: The Two Towers, for which he received his first Oscar for best visual effects. During his 12 years at Weta Digital, Letteri has supervised visual effects for 14 films, of which eight resulted in Oscar nominations for best visual effects and four received Oscars (Avatar, King Kong, and Lord of the Rings: The Two Towers and Return of the King). In addition, Letteri received a Technical Achievement Award from the Academy in 2004 for his part in developing subsurface-scattering techniques for rendering skin.

Letteri shares his nomination for The Hobbit: The Desolation of Smaug with animation supervisors David Clayton and Eric Reynolds, and visual effects supervisor Eric Saindon, all of Weta Digital. The film, which has grossed $848 million worldwide, also received nominations from the Visual Effects Society for outstanding visual effects, virtual cinematography, FX and simulation animation, and compositing — and it won the VES award last night for outstanding animated character in a live-action feature.

StudioDaily: You have received eight Oscar nominations now. Is the news always as exciting as it was the first time?

Joe Letteri: Always is the key word. It’s a good way to wake up in the morning.

The 10 films up for nominations all had great visual effects. Why do you think your peers voted for Hobbit?

Middle Earth is a great place to make a film, and I like the way this film moves. There’s always something fantastic as you move through the landscape, and the action keeps going from piece to piece. I like the way Peter [Jackson] interleaves multiple story lines that keep the film interesting. He takes world-changing events and reduces them to the individual drama, to something you can read on the actors’ and characters’ faces.

The thing we’ve always been fortunate with in these films is that because of the way Tolkien wrote the books, we are able to find characters that let us do something that couldn’t have been done before. In the last film, we had a slew of characters, but also Gollum. We brought him back in a great performance piece that helped give the movie a bit of focus and took it above the other work in the film.

In this film, we have 60 characters. You can’t wrap your head around the fact that they are not real — they’re digital. But with Smaug, we suddenly have this big talking character that seems sort of human even though he’s a dragon. And we have this nice dialog between him and Bilbo, two actors working off each other but with this magical look. That always interests audiences.

What makes Smaug a successful creature?

I consider Smaug a character, not a creature. A creature is something you react to. Smaug is something you engage with. The other character [Bilbo] engages with him and the audience engages with him.

That’s what I liked most about Smaug — trying to understand that connection to a human performer. We looked at the scene in the treasure hall like the riddle scene with Bilbo and Gollum. We wanted that dialog, drama, and rapport between Smaug and Bilbo. In both, Bilbo is playing for his life.

But all the other characters we’ve done, from Gollum onward, have been humanoid, so we had that touchstone to understand their performance. Caesar [from Rise of the Planet of the Apes], King Kong, even the Goblin King [from the previous Hobbit film] were humanoid characters. Smaug is our first non-human, but he still had to intuitively think and look as a human. That was the most interesting part. What do we use to convey human emotion on that lizard / alligator face?

Did you use performance capture for Smaug?

Benedict [Cumberbatch] was on a soundstage, not in a booth, so he could move around and Peter could direct him on a stage, and we put him into a suit for the first days. He tried some different things – slithered, lifted his neck up – that helped us understand what he was thinking about the dragon. But a human can’t shift his weight in the way a dragon does. So it was like storytelling. The animators could look at video footage of his performance and say, “We get it.” We could see where he was being secretive, holding his thoughts back, being paranoid.

It was interesting to convey the emotional cues we’re used to seeing through the dragon’s performance. We adjusted his wings to give him a sixth finger, to give him hands to gesture and grab with — he has hands at the end of his wings with a thumb like a bat. We articulated his eyebrows and used his tail to convey body language. His tail telegraphs when he’s about to strike or relax. We were looking at ways to bring out his personality.

Was it a treat for the animators to perform a character entirely with keyframe animation?

Even when we have performance capture, they animate a lot over and above.

Azog, performed (via motion-capture) by Manu Bennett. Photo courtesy Warner Bros. Pictures

Did you use motion capture for other characters?

Bolg and Azog, the Orcs, are always motion-captured because they’re fully CG characters. When we shot the film, we had the guys in makeup and prosthetics, but Peter [Jackson] wanted something more organic. When Bolg shows up in the film, he has a helmet hammered into his head. There was no way to do that with makeup or prosthetics. We kept the live action only for the Orcs at the sluice gate, and we replaced their heads. Otherwise, we replaced all the Orcs in the shot. Because of the Dwarves, we needed a scale difference, so we shot Orcs in costume and then replaced them with [digital] Orcs the right size. We used a lot of motion capture in the barrel chase down the sluice, especially when the Dwarves are in the water. We had digital doubles for all the Dwarves.

Did digital doubles play a major role elsewhere in the film?

We had a couple shots of Legolas in the water during the barrel chase. We shot [Orlando Bloom] against green screen for a lot of the shots, but when Legolas is leaping from bank to bank, he’s digital. Tauriel [Evangeline Lilly] makes a cool entrance as a digital double. It used to be we’d use digital doubles mainly for the heavy action scenes, but as they have gotten more and more refined, we’re bringing them closer and closer to camera. And, in the past, we’d shoot actors on green screen for big, wide reveals and then shrink them and put them into place. Now, we use digital doubles.

In what way are you refining the digital doubles?

Creating a believable human is still a technological challenge — how the lighting works with the skin, the muscle simulation, the facial simulation, hair and cloth dynamics. All these simulations still require a lot of handwork. None of this is automatic. There’s really a lot we don’t know. But as we chip away, we understand more about how to do these things and we lock a few more down. We still get in and do work on top, but each time, the base level is a little bit higher.

Tell us how you approached the barrel chase.

We started with artwork from John Howe and Alan Lee and then previs’d that whole chase using the ideas Peter [Jackson] wanted to hit. We stitched the previs, the location photography, and the artwork into a rough layout to understand, if the barrel chase were real, what it would be.

Location photography?

Twice a day there is a controlled release of water at the Aratiatia dam. It’s dry until they release water, and then there are these amazing rapids. So we could scan the empty landscape and figure out what the layout would be. We worked out a two-kilometer stretch of river and then broke that into bits. When they released the water, we put empty barrels into the water and put digital dwarves into those.

We also had Steve Ingram, our special-effects supervisor, build a ring of water on stage and we floated the actors in barrels for close up shots, reaction shots. When we needed bigger action, sometimes we’d put the actors in barrels with wheels on a green screen stage, and they’d walk around and pantomime.

Was the water always live action?

We had digital water for about a third of the two kilometers, about a 700-meter section. In total we ran 76 simulations. We moved an average of 20 tons of digital water per second of film through this intricate waterway and ravine.

The simulations had to be fairly huge to get enough water flowing through. And, of course, every time we did a simulation, we had 13 or 14 barrels and people jumping into the water. We had cloth inside the barrels and if the cloth simulation didn’t work, or we’d have to fix the animation of the dwarves, we had to re-sim the water. It was interactive, and so complex. It took a lot of hand work to push it through. But in the end, we learned a lot about water simulation.

Did you do anything differently in this film to help match the digital and the real water?

In the past, it has been standard practice to break out components in a simulation for rendering. Here’s the water with some absorption and some color, here’s the foam on top, here’s the spray. But it’s really a continuum. It’s never just particles. There’s volume. The hard thing has been to render the aeration under water the same as on top. This time, we treated it all as one continuous thing and rendered it all in the same simulation. Now when we’re doing ray-tracing into the water, the ray hits the foam on top and moves through. We have refraction on the droplets in the air and the bubbles in the water. There is a continuum of water. We still broke out some spray for interaction, but the base water simulation was rendered all in one.

How big is the render farm at Weta Digital?

We’re in the tens of thousands of processors. I don’t know exactly how many, because it’s dynamically configurable. We put through 200 shots a night. Not all finals, but a fair amount of work. With 48 frames a second, stereo, it adds up.

Do you feel like being able to work on Peter Jackson’s films is a luxury?

We don’t get all this handed to us. We have to make the case that a film like this deserves the investment. We want the quality and an audience experience that will hold up for a lifetime. People tell me they watched the trilogy with their family. We’re thinking that way, that these films are things people will put in their library and pull out over the years, so we want the films to work as well as they can possibly work. It does require a big team to do that, and, yes, it helps that we can do all the work at Weta. We don’t have to spend time coordinating. We can put our time into developing the art and science we need.

How many people worked on the film?

We had 1000 artists and 2000 shots.

When we did the Rings, we started with 150 people. The first Rings had 450 shots. By the third, we were up to 1500 to 1800 shots, but we had a lot of practical effects and miniatures. These days, even when we’re shooting plate photography, we’re augmenting a lot of it digitally.

In the first Hobbit, we kept a lot of the landscape as is. In this one, we’re in fantasy realms very soon — the Mirkwood Forest, the Elven Kingdom, the barrel chase down the river, then Erebor. Even Laketown – there were set builds, but we had to extend all of them and we had to create exteriors.

How do you see visual effects moving forward?

For me, personally, the big step was coming here to do Gollum. Before that, we did creatures. If we did a character, we tended to puppeteer it. We could do dinosaurs, but they were creatures. Gollum was the first character that had to work for the film. I remember in Two Towers, Gollum had a major role and he wasn’t going to be Andy Serkis in costume. If that didn’t work, the movie wouldn't work. He wasn’t the main character, but he was a big focus. So for me, that was the breakthrough: Bring a character that’s almost human to work side by side with actors. For the last 12 years or so we’ve built on that.

Anything you need to make a movie, we do digitally these days. We’ve been fortunate that the films we do focus on character. King Kong, Avatar, Tintin, Planet of the Apes, Hobbit. Each film has a strong digital character, so we’ve kept that as the focus while at the same time developed world-building facilities — jungle, landscapes, mountains. You kind of need to develop both capabilities. But for me, it’s keeping the focus on the character.

Do you ever get tired of this – all these interviews, all these questions?

No. It’s a nice rhythm to put all your effort and focus on your work, get it done, and then talk about it afterwards and think it through.