On Live-Action VFX Workflow for Animation, Making CG Look Real, and the Largest Stop-Motion Puppet Ever

The Oscar nomination for visual effects in Laika’s stop-frame animated film Kubo and the Two Strings is Steve Emerson’s first. He joined Laika in 2008 to work as a 2D supervisor for that studio’s Coraline, became compositing supervisor on ParaNorman and co-VFX supervisor on The Boxtrolls. Those three films all received Oscar, BAFTA, and Annie nominations for best animated feature. Prior to joining Laika, Emerson worked as a digital compositor on The Matrix Reloaded and The Matrix Revolutions, Transformers, and Beowulf, and has credits on 25 films. He became VFX supervisor on Laika’s fourth film, Kubo and the Two Strings. Directed by Travis Knight, Kubo and the Two Strings is also nominated for Best Animated Feature in addition to its visual effects Oscar nomination, Annie and Golden Globes nominations for best animated feature, and a VES nomination for outstanding visual effects in an animated feature. Kubo won the BAFTA film award for best animated feature. 

Steve Emerson shares the visual effects Oscar nomination with Laika’s Oliver Jones, Brian McLean, and Brad Schiff.

​Kubo and the Two Strings VFX featurette

Studio Daily: Why do you think your colleagues in the VFX branch voted for Kubo and the Two Strings to receive an Oscar nomination?

Steve Emerson: As much as this film is a tribute to the Japanese culture and to Japanese woodblock artists like Kiyoshi Saito, it is very much an homage to special effects artists like Ray Harryhausen, Willis O’Brien, Jim Danforth, and the many special effects artists who laid the foundation for the entire industry with the innovations and art form they created, which is stop-motion animation. I think there is an understanding and appreciation for that type of filmmaking within the VFX branch. The skeleton we built for this film was an homage to Ray Harryhausen and Jason and the Argonauts. It’s the largest stop-motion puppet ever created.

When I was young, there was such a sense of awe when you would see something extraordinary, a sense of how did they do that? If you go back 20 or 30 years, pre-computers, if you wanted a monster, you built a puppet or put someone in a suit. It was movie magic. Now, my kids see something extraordinary and it’s largely dismissed as computers. So I think there is an appreciation of who we are, that we’re hanging on to the techniques of the past and trying to respect, honor, and innovate upon what I see is the core of movie magic.

I’ve heard people say that visual effects nominations should only apply to live action films, not animated films.

When people think of animation, they often think of 2D cel animation and CG animation, but the reality of the work we’re doing at Laika is live-action filmmaking. It is a live-action visual effects workflow. We’re making live-action films and doing live-action visual effects, but in a way that’s extremely difficult and takes a lot of time.

How is your workflow similar to a live-action visual effects workflow?

Our actors are very small, and we film them one frame at a time. But those actors are on sets and on green-screen stages. If you were shooting a kung fu scene with live-action actors, you’d attach the actors to wires and put them on a green-screen set, and shoot them at 24 frames a second. We attach complex rigging to puppets, put them on a green-screen stage, and shoot them at a rate of three to five seconds a week. The guts of the process is the same. It just takes longer and, because it takes so long, a lot of things can go wrong. We can’t do three or four takes of something.

We’re using a lot of the same software a visual effects artist would use for a Marvel movie. We use Houdini. We composite in Nuke. We do a lot of 2.5D compositing. We do effects — smoke, fire, rain, water. But we have to go through a whole other level of design to make sure these components feel handcrafted and fit into the aesthetic of this world. If you need an ocean for a Marvel film, you open up Houdini and get to work creating an ocean. We have to make oceans that look like they could have been made from garbage bags and were hand-animated, fire that was crafted from cheesecloth, lit and reflected through shower glass. It’s extremely complicated work.

But it all begins with our stage plates and our performances and that’s no different from live action. We make hybrid films.

What do you mean by hybrid films?

There’s a great example at the front of the film where Kubo says goodbye to his mother. It’s a big, wide helicopter shot. The camera pulls back and you see Kubo’s mountain with water around him. We have full-scale hand-animated puppets on a full-scale ground plane combined with a miniature set, a miniature mountain, miniature rocks, digital water, and a digital sky.

We also do digital set extensions. When we build digital assets to extend environments, we use the materials from the physical sets as reference. For example, there’s a big wide shot when Kubo walks into the village at the beginning, when he does the initial performance with origami. We had a foreground plate, the bridge he’s crossing, and some background buildings. The rest are digital assets. And, for the showdown with the moonbeast toward the end, the immediate surroundings where they’re moving up and down the main street in the village are largely physical with buildings on stages. But beyond that, the buildings are digital extensions. 

If we build a mountain, it’s not just a mountain, it’s been crafted from a specific type of material and has a specific shape language relative to design. So, we scan small-scale maquettes from the art department to create the digital assets. 

Are any of the villagers digital?

Our hero characters are always hand-animated puppets and there is a smattering of practical puppets but the extras are, for the most part, digital. When we create a digital asset or extra, it’s never done in a vacuum. We collaborate with all the departments at Laika. We receive designs, physical materials, and reference photography from the puppet and the camera departments. The same leads for the physical puppets come to the visual effects department and work with us deskside as we build the digital puppets. We don’t show a puppet to the director until the costume, hair, and puppet looks like it represents a physical puppet’s style. Then we loop in the animation team. The digital animators are given notes by the supervisor overseeing the animation on stages. You can do so much digitally. It’s infinite. But there are so many limitations to stop-motion in-camera photography. We have to respect those limitations. It’s a huge advantage for us to have that level of collaboration and access to these people.

Was the Moonbeast digital or practical?

It’s the first entirely 3D-printed puppet. It’s made from over 881 different parts, all created on 3D printers and assembled by the puppet team. They used a gooseneck microphone stand for its spine. Then it was a big collaboration across the departments to deliver the final look. We shot it in multiple lighting setups using UV paint and black light so we could extract details and enhance it in post production.

Does the visual effects team also paint out the lines on the puppets’ faces?

There is a ton of cosmetic work. About half the team in visual effects is doing the 2D work — rig removal, compositing, and painting out the lines.

There is a split line across the center of the faces that needs to be painted out in every frame. And with Kubo, we also decided to have the cosmetic team paint out by hand the separation between the eyelid and the face. This is new. When you look at a stop-motion puppet, the eyeballs with eyelids are attached to the inner workings of the head. The face parts snap over the top and register with magnets over the top of the eyeballs. If you look at previous films, there is a small separation between the eyelid and the facemask. Painting that out gave this film a distinctive look. It felt oddly real and humanistic. Ultimately we are trying to create empathy with the audience, to take people to a place where they forget they’re looking at puppets. This is done with subtlety using an incredible amount of variation in the facial performances. Kubo was capable of 48 million facial expressions. Painting out the separations helped make the performances as realistic as possible. 

What was the hardest sequence for your team?

The storm with the puppets on the long lake. The water is getting choppy. The camera cuts in on a shot of the boat. It’s moving forward quickly toward camera, almost like a surfboard, and overtakes the camera. The boat is practical for the entire sequence and the water is digital. We started on it during the first month of production and it went on for a good two years. It’s technically challenging. We’re very close to camera. There is a lot of interaction and specific performance-based notes between the boat and the water. 

On every one of these films we create a piñata at the end that’s a tribute to the hardest shot in the film. In the case of Kubo, that was the shot. The production team built a piñata that looked like Kubo’s boat. We all went into the parking lot and hit it with a baseball bat. Little airline booze bottles fell out, and we had a party.

What part did the other nominees play?

Brad Schiff is the animation supervisor for all the stop-motion animation and he worked with our animation lead on the visual effects side. Brian McLean is the rapid-prototyping supervisor who leads the 3D printing team. Oliver Jones is the rigging supervisor. He set up all the animation rigging, and his other thing was that whenever we needed in-camera effects to set the look of a visual for the digital side, he would figure it out. He did all the concepts for our digital work on water and fire. We digitize what he builds and build upon that. Once we have the digital asset, we can propagate it across 100 shots.  If it makes more sense to do a digital interpretation, he sets the foundation. If we’re going to shoot it in camera, he oversees that. He’s like a special-effects supervisor and concept artist.

When we received the visual effects nomination, the idea was that it wasn’t only the digital work being honored, it was the studio as a whole, the collaboration, the stop-motion animation. Every frame of these films, every frame of a performance, it’s all a visual effect.

What did you learn from this film?

At the outset, Kubo and the Two Strings truly felt impossible in terms of scope, the amount of work, the innovation, the things we’d never done before. It didn’t seem like we could do it. But the artists came through in such a huge way.  Our team has hardly grown since Coraline, [but] the stories are bigger and more complex. And we’re still getting it done. I learned that I can count on this team to consistently deliver regardless of how Herculean the task appears to be. At this point, I feel they can throw anything at this team regardless of scope and complexity.

Do you know if this is the first stop-frame animated film to receive a visual-effects nomination?

Coraline made it to the top 20, but not the bakeoff. So, this is the second one to make it to the top 20, but the first to go all the way to the bakeoff and the nomination.

How did you react when heard you received the Oscar nomination?

I was overwhelmed. I was sitting on the couch with my wife and my son and I started crying. I didn’t know it, but my wife put a picture of me crying on Facebook. People at work said, “We saw you crying on Facebook.”

We spend years and years working on these films and the artists I am fortunate enough to collaborate with pour everything into creating these films. So when the nomination happened, it was very emotional for me. I was very happy. It felt like all the work was finally being recognized by the visual effects industry on a whole other level. We’re a small independent studio in the Pacific Northwest and to be given that level of recognition is so overwhelming. It was like I was in the basement cutting an album on my banjo and now I’m up for an award with Dylan and McCartney. 

Featurette: "Choose Kubo"