Gollum, the Goblin King, and Making Fantasy Figures More Realistic

Weta Digital’s Joe Letteri has won four visual effects Oscars (for the second and third Lord of the Rings films, King Kong, and Avatar), and has received seven Oscar nominations—including this year’s nomination for Peter Jackson’s The Hobbit: An Unexpected Journey, a New Line Cinema production released by Warner Bros.

Letteri began his career at Industrial Light & Magic, where he was a CG artist on Jurassic Park, associate VFX supervisor on Mission: Impossible, and vfx supervisor on Magnolia. He joined Jackson’s Weta Digital to supervise the visual effects for the 2002 film, The Lord of the Rings: The Two Towers, and earned his first Oscar for best visual effects. In 2004, he received a Technical Achievement Award from the Academy for his part in developing subsurface scattering techniques for rendering skin. Letteri is now Weta Digital’s senior visual effects supervisor and the current director of the firm. He shares his nomination for The Hobbit with animation supervisor David Clayton, and visual effects supervisors Eric Saindon and R. Christopher White, all of Weta Digital. The film, which has grossed $956 million worldwide, also received Oscar nominations for makeup and production design.

Studio Daily: Why do you think your peers voted for The Hobbit for visual effects nominations?

Joe Letteri: With The Hobbit, two answers. The scope of the work. And the primary reason, I think, is the characters. I like to focus on the characters. We had six characters with dialog scenes and speaking lines, not to mention all the other characters and the landscape.

So . . . Goblin King or Gollum?

For me, it’s still Gollum. He’s my favorite. The performance – especially the way Andy [Serkis] plays him is so complex. We see his dual personality. Even in a nine-minute scene, he goes through a character arc from being completely in control, to taunting like a cat with a mouse, to completely losing it. Plus, it was one of my favorite scenes from the book when I read it. To bring Gollum back, we reinvented him. Everything that went into him is different: Skin, muscles, hair, eyes. We gave him more of a sense of realism.

Why is realism so important with a fantasy character?

I think it’s very engaging. To me personally, and to audiences as well, when you look at it, you think it’s real, but you know it can’t be, but your brain keeps telling you it is. It causes you to focus in a way you wouldn’t do otherwise. Gollum is a good example. If he were an actor in makeup and a wig, I don’t think you’d focus on him in the same way. He’s realistic to the point of hyper-real, a creature that can’t exist, but you know it does. When we do fantasy, it always has to have an element of realism. If you violate any basic principles—weight, the way the eyes work, the way the skin works—any of those things will break the illusion.

You want to take as much real as you can and add the one thing that’s unique. Gollum looks pretty much like a person, but his head size, his eye size, can’t exactly fit into a human form. But, the way he moves and speaks, the way his skin moves, it’s as if he’s a human. You get the best of both worlds. He’s grounded in something believable so your mind accepts him, and he’s a fantastic thing you’ve never seen before, so your mind is engaged.

Why did Gollum need a remake?

When we did Gollum the first time, we approached him like a digital version of an animatronic. We built a form and stuffed it with things that make it move. He had an articulated joint system that was almost like a digital armature. What we were calling muscles were inflatable bladders. That’s like what you do with armatures.

But since then, we’ve rebuilt him so the biology is correct. It isn’t like there’s an armature; there’s a skeleton with bones in the right shape and density. Attached are muscles that are fiber bundles surrounded by a fascia and tissue layer and then we have fat on top of that and skin on top of the fat. It’s all solved from the skeleton outwards. The physics are the same. If you look at Gollum in the riddles, you can see his muscles moving under his skin.

You’re adding physics to the character animation?

We’ve gone away from making skin that looks like cloth. We create it as an organ and solve it like biology. It’s akin to a surgical simulator. We don’t think in terms of rigging; we think in terms of biology.

Why do you use motion capture so often for human characters and animals?

We use it because it’s physically real. We get a sense of weight and micro movement we can’t get any other way. Animators can figure out how to do a certain gesture, say a hand reaching to pick up a ball, and work out that style of animation. But, if you’re a human, you would never do that motion the same way twice. The muscles always take a different path. You never get repeatability; there’s always something unique. It will have the character of the actor, but it will be unique.

Golllum might be your favorite, but I think the Goblin King is getting more attention. Any idea why?

[laughs] Because he’s gross. People like looking at things that are gross. Technically, it’s all in there. The performance is great. He has this huge goiter. He’s covered with boils and pus and has this weird diseased skin. That’s the fun of the character. You’ve got these little goblins, wiry scavengers. The King dominates by virtue of his size. But he has no regard for anyone’s life other than his own. He’s completely disgusting.

Did knowing you would be working on three films helpful?

It helped us be able to pin the R&D. But, as far as the scope – from Avatar on, the films we’ve done have been so wide in scope that we have to keep R&D going on simultaneously. In Hobbit especially, because it is [stereo] 3D, things that might have been elements before had to be 3D – like flames, clouds, waterfalls, landscapes. And then we had character and creature design not only for the fantasy characters, but digital doubles. The R&D is still going on for all fronts, but knowing about the next films gives us some specific things to work on. We know we’ll need to do multiple layers of clothing with heavy, stitched leather. Details in eyes. We can focus on those areas.

What was the impact of 48 fps on post-production?

In a way, it was as simple as twice as many frames, so we had to do more work. It did allow us more creativity with animation. When you have 48 frames for every second, you can handle quick changes of motion better. You can see that in Gollum. At 48, you can really define those micro expressions. At 24 fps, the expressions are softer. We capture at 60 frames per second, so we could use more of the motion-capture data.

Did shooting in stereo eliminate the use of forced perspective?

There were times when we could get away with it. Often, say we had Gandalf and Thorin, we’d roto Gandalf off the plate, blow him up, and lock him into place. That worked as long as we could track the movement. If he was walking, that didn’t work any more.

There’s a scene in Bag End, a minute long. Gandalf is putting silverware on the table. They had Bag End fully dressed, full-sized, with markers in the ceiling where Gandalf was supposed to be. But, next to it, Sir Ian was on a green-screen stage, smaller in proportion. The encoded camera in Bag End drove the motion-control camera on Ian. He had C-stands with the actors’ faces, so he knew who was talking to. All the actors had ear pieces. It was all choreographed live and rehearsed. Peter and the camera operator saw both images composited.

We were able to get that shot in four takes, but it took 18 months to complete it by the time we got everything rotoscoped and finished. In [stereo] 3D, it’s easy to have one body float through another.

It sounds like a variation of virtual production. Did you use virtual production techniques throughout filming?

We did all that. We captured Andy live with Gollum. One to one. We had sets with the trolls with a motion-capture stage adjacent to the live-action stage. Peter could direct, and the camera operators and Peter saw the trolls mixed with live action. Everyone could see what it would look like. In Bag End, we used the same technology to drive the two live-action stages simultaneously. I guess that means we’ve almost come full circle.

Couldn’t you have done some of the shots entirely digitally?

If we shoot and it works, great. If it doesn’t, it’s great for the edit. Sometimes, especially with the [stereo] 3D, it’s easier to replace than salvage what was shot. It’s interesting that we’ve gotten to that point. We had shots inside the goblin caverns that were really complex. At first, the goblins were going to be actors in full costumes. That lasted 20 minutes before the heads came off. It was too hot and they couldn’t see. OK, so we’ll replace the heads. We knew where this was going. Once we had to replace the heads, we also had to track the bodies. So, we painted out the bodies, used performance capture, got new performances, and put them in 3D. That happens more and more. We’re starting to treat these things like we’d better be prepared to do the shot all-digitally.

It sounds like you had the great luxury of enough time and budget to do the film.

[laughs] I wouldn’t say that. Remember, we started out doing two films for The Hobbit. No one has done a duology. And around August, it turned out that, yes, we would in fact do a trilogy. We thought, OK, great, we can spread the story out and move a few hundred shots into the next movie. And then Peter came back and said that we needed to add this whole prologue. And suddenly we had 1000 new shots with six weeks to finish. So it wasn’t as luxurious as it might sound.

I noticed that people from Weta Digital received visual effects Oscar nominees for three films this year. I think that’s only the second time – since ILM got three nominations in 1991.

What’s interesting about it is that [Hobbit] is the only show where all the work was done by one studio and in one place. On all the other films, the work was split up. If you look at the engineer’s sequence in Prometheus, the tissue simulation and subsurface scattering are the same techniques we used for Gollum, the trolls, the goblin king, the digital doubles, and also the digital doubles in Avengers. We had six main characters in The Hobbit, but we also had digital doubles in the other shows and three main characters in Prometheus. It was the same with Rhythm & Hues. They did beautiful creature work for Snow White, but it’s attributed to the film, not the studio.

The recent news that Rhythm & Hues is filing for Chapter 11 protection is discouraging. 

It was a shock. We thought they had it sorted. The problem we’ve all got [is that] a visual effects facility has a big overhead. On some of these films, we are the largest component. We had 1,000 people working on The Hobbit and a lot of technology.

A lot of studios don’t realize that because they don’t see us. They see four visual effects people [on set]. They ask why we take all that time to put up a green screen, but to us, holding up 200 people is nothing compared to holding up 1,000 people later on. Until people from the studio go to the buildings and see how many people are working on their movie, they won’t see the impact of the decisions they make.

It’s the same with booking a show. To make a release date, we have to start hiring people, often, before contracts are in place because we know that otherwise we won’t make the date. You think a project is going ahead. You commit to people. Then the work is delayed or goes away and suddenly you have hundreds of people that you’re responsible for and no work.

It sounds like scheduling is causing big problems.

Exactly. Really more than anything, it does come down to that. Films get started all the time and get shut down. But most studios seem to think they’ve committed to only a few people. They don’t realize that we’ve had to commit to hundreds of people on the other end. I don’t have the answer. But there needs to be more discussion. They need to know what visual-effects studios are carrying to make their films happen.

Leaving the business trends aside, what trends do you see technically and artistically in visual effects?

Generally, I see more of this trend toward realism. In a way, that’s what we’ve always done. But now, there’s more acknowledging that it is what we do. There’s more of a focus on understanding and trying to apply realism. Even though it’s more complex, it gives you the ability to standardize around a known quantity. There is less guesswork when you measure the real world.

What are you excited about now?

Smaug. He’s our next big character. You just got a few glimpses of him in the first film. I love the Riddles in the Dark, and I love Smaug. Seeing what we can do with Smaug is the next thing.