On the Film's Massive Cityscapes, Choreographing a Threesome with a Replicant, a Human and a Hologram, and Blade Runner's Rachael Reborn

John Nelson’s visual effects Oscar nomination for Blade Runner 2049 is his fourth overall. In February, he received BAFTA’s Best Special Visual Effects Award for the film, and he has received an additional two previous BAFTA nominations.

In 2001, Nelson received an Oscar for Gladiator. He has also received Oscar and BAFTA nominations for Iron Man, an Oscar nomination for I, Robot, two VES nominations for best visual effects and best single effect award for Iron Man, and a BAFTA nomination for Gladiator.

Nelson began his career at Robert Abel and Associates (as did other Oscar winners including Rob Legato and Charlie Gibson) as a cameraman and a technical director. After working in Berlin to help set up the German company Mental Images, Nelson returned to the US to work for Industrial Light & Magic. He has a film credit as a CG animator on Terminator 2. He became a VFX supervisor at Rhythm & Hues for his next film, 1992’s Stay Tuned, and then moved to Sony Pictures Imageworks where he supervised effects for a number of films including City of Angels and Anaconda. He became an independent visual effects supervisor to manage the overall effects for Gladiator, the first of several films with multi-million dollar budgets to follow. In addition to his award-winning stints supervising I, Robot and Iron Man, he is credited on 21 other films.

Blade Runner 2049 has received five Oscar nominations (visual effects, cinematography, production design, sound editing, and sound mixing). The film received BAFTA awards for visual effects and cinematography along with additional BAFTA nominations for music, production design, sound, editing, makeup/hair, and direction; two VES awards (LA environments and LAPD model); and five additional VES nominations (animated character, Vegas environment, Trash Mesa environment, compositing, and best visual effects).

Directed by Denis Villeneuve, the Alcon-produced and Warner Bros/Sony Pictures-distributed film earned $259 million worldwide and achieved an 87 percent positive critical rating on Rotten Tomatoes.

In addition to John Nelson, Framestore’s visual effects supervisor Richard R. Hoover, Double Negative’s visual effects supervisor Paul Lambert, and special effects supervisor Gerd Nefzer received Oscar nominations and the BAFTA award.

Framestore just posted this VFX breakdown of its work on Blade Runner 2049.
Framestore/Youtube

Studio Daily: Why do you think your colleagues voted for Blade Runner 2049 to receive Oscar nominations for Best Visual Effects?

John Nelson: I think the film’s restraint made it stand out, and that’s a credit to [director] Denis [Villeneuve]. Visual effects can sometimes be an over-stimulation explosion. When I first interviewed with Denis I asked him if he wanted belching fire and industry pollution like the original film. I’m from Detroit, so I know about post-apocalyptic landscapes — fire out of smokestacks. He said no, he wanted almost a silver landscape with claustrophobic atmosphere. Like Montreal in February on a bad day. So we consciously pulled things back a bit. Denis wanted the effects to be in the fabric of the movie, to have an emotional reason to be there. I think people chose the film because of that.

And because of the wide variety of effects. We had effects from cities to humans. Also, Rachael and the merge [Joi and Mariette]. Everyone knows that producing a digital double that not only looks real but can act is a big deal in the visual effects world.

How many studios worked on the film?

We had eight vendors, and I was blessed that I had the team I wanted. The biggest were Double Negative in Vancouver and Framestore in Montreal. I have a theory that no vendor should have more than 250 shots. When they get up to 1,000, inevitably there are some shots they like and some they don’t like doing. So, I kept the shot count down. DNeg had 281 [and] Framestore 271. But their sequences were very difficult.

We had Framestore’s art department in London and DNeg’s art department, too. Buf in Paris and Montreal. Buf Paris did Elvis and Montreal did Anna’s memory lab. UPP did the hall of replicants. MPC did an outstandingly good “Rachael reborn” for our film that brought Sean Young back as she looked in the 1982 original Blade Runner. Atomic Fiction did the bar with the giant ballerina. Rodeo’s art director Deak Farrand did the great matte paintings for the orphanage scenes. They feel so warm and wonderful. Weta Workshop built the miniatures for LAPD.

A desert cityscape in Blade Runner 2049
Alcon Entertainment

Did you use sets and miniatures rather than CG?

Denis, [DP] Roger [Deakins] and I did not want the movie to feel like it was all CG. We wanted it more real and restrained and dirty. Not shiny at all. Grimy and lived-in. So we looked at every scene and built as much set as we could afford. And we wanted to use matte paintings and miniatures as a way for it to not feel CG. We had a back lot in Budapest as big as a football field. We put one trash mesa and one Sapper farm set on it. But when you start shooting big wide shots, they’re not that big. So every wide shot in the movie is a visual effect.

How many visual effects shots are in the movie?

We knew we’d have more than 1,000 and had thought it would be 1,400. We ended up with around 1,200. The film is two hours and 43 minutes. There’s one hour without visual effects. Denis is fond of long shots, and he doesn’t shoot a lot of coverage.

Tell us about building a restrained city.

Ana [Carla Juri] has a line in the movie where she tells K [Ryan Gosling] what makes memories real. She says that people think it’s about detail, but it’s really about emotion. I thought, yeah, that’s the same with visual effects.

You need enough detail in a city to assume it’s real, but once above that, it’s about where to look and the emotional intensity. A city is made from hundreds of millions of decisions made over thousands of years, and we needed to have that chaotic variation in design to make it feel real and not built by one person. Then, we needed to populate it with signs and advertisements, but not so much that it was overloaded. Everyone has seen movies that emulated the first Blade Runner, where they’ve added everything and the kitchen sink. We didn’t want our city to look CG or look like an overstimulated visual effect. If something drew attention to itself, we knew we didn’t want that. So, we worked on atmosphere, subtlety, and fit, and pulled everything back a little.

Joi (Ana de Armas) and K (Ryan Gosling) on the roof in Blade Runner 2049
Alcon Entertainment

How did you create the atmosphere you wanted?

It depended on whether we were working on top of a city plate. Roger [Deakins] had done a first grade for shots on stage and the backlot, so we re-graded our plates to fit into that, and those re-graded plates were the basis for the big scenes.

When we fly from Sapper’s farm to the LAPD, that’s Mexico City until we get to the Weta Workshop miniature. We added a ton of CG atmosphere — CG snow, signs, and holograms on top. The final finishing sauce was Z-level atmosphere pushing back into the distance so we’d see things coming out of the deep fog. It was a tricky balance. We had to judge how much depth we could have in wide shots and sometimes we’d cheat. Otherwise we’d only see back five feet.

For the shots over Mexico City, we scouted for when we would get cloud cover. Roger and I agreed that the perfect light was an overcast sky, slightly backlit. He was adamant about shooting in soft light because you can’t make hard light soft. So, while we were shooting plates in Budapest, in my spare time I would scout areas in Mexico City in Google Earth. We knew which months would be best for clouds. When Dylan [Goss, aerial DP] went to Mexico, I sent him my Google Earth fly-bys, complete with GPS coordinates, of the areas that I thought would look good for shots we storyboarded and previs’d.

Our hardest shots were the cities, K’s holographic companion, Joi, and bringing Rachael back as she looked in 1982 to perform in our film.

What made shots with Joi difficult?

It was hard to get our heads around Joi to keep her in the spirit of the film. We had five things that had to happen simultaneously for Joi to work: subtlety, realism, to not look like CG, to not call attention to itself as effects, and to fit the story. We looked at all the holograms that have been done and didn’t like them. They were too glitzy.

So how did you create Joi’s hologram effect?

We shot the actress, mapped on a CG surrogate, and created a digital back face. When she’s backlit, you can see through her front. It’s like when you look through a glass of water and see through the front to the back. So, real photography provided the realism and the actors performance and the CG back shell provided the depth.

We shot Ana de Armas with Roger Deakins’ ARRI Alexa but also with witness cameras all over the room so we could see her from every direction — four GoPros and a Canon C300. We modeled her in CG and positioned the geometry in her exact position and then we projected her performance onto the CG model.

By the time Joi walks out into the rain, we wanted everyone to know who she is and how she works. The rain fritzes her out a little bit. Then her software says, oh, this is rain, learns about it, and emulates it. As a spinner goes through the scene, you see its light rake through her and you see her back shell.

Double Negative released this VFX breakdown highlighting its work on Blade Runner 2049, including the “merge” scene.
Dneg/YouTube

There is an amazing scene with Joi when she merges with Mariette. How did you make that work?

We shot the two actresses separately, both with camera moves but without motion control. The room we were shooting in was incredibly small and we couldn’t be in too many places without going in front of Roger’s lights, which is not a good place to be with Roger. So we had GoPros in every corner of the room and a witness camera off to the side of the taking camera. We shot Mariette [Mackenzie Davis] first. Denis would pick the take. Then we’d break down that take. At two seconds, she lifts her hand. At four she touches him. At eight, she walks around him. And so forth. Then, we put an iPad with those shots in front of Ryan’s [Gosling] face so he could line up his eyes. We pulled the iPad out, and fed Ana [who plays Joi] the directions. OK, now raise your hand. Now walk around him.

Even though we tried to direct the actors as little as possible we’d get moments when our double exposures lined up like magic. [Editor] Joe Walker cut the performances together and found the best way to balance the moments. We had some moments of greatness and some not so great.

Then we went back in, and for every shot we did three weeks of tracking and mapped both women onto their CG surrogates. We moved the surrogates for balance and then mapped the two together in full CG. There’s a moment when Mariette says, “Wow, look at you.” You see more of her, and then we shift to Joi and Joi says, “Quiet now, I have to sync”. She looks up, looks down at K, and the two women merge together. The mix, the third woman, looks different than either. Then that merged third woman throws a look at K. I worked with Paul Lambert at DNeg on those shots. I was worried that the scene would peak too early, so we had a series of shots that would start out of sync then be in sync by the end of the shot. That way, the sequence would build. When we got that merge at the moment K pulls her toward him and their eyes line up and she looks at him, I shouted, “Wow, yes, yes, yes!” and leapt out of my seat.

You said that Rachael was also difficult. How did you so convincingly reproduce the replicant Sean Young played in the first Blade Runner?

We shot Loren Peta in full make-up and hair, acting with Harrison Ford [who plays Deckard]. Sean Young was there as an adviser sitting next to Denis [Villeneuve] while we were shooting. She calls Harrison “Harry.”

Roger had designed a circular light that was always moving. We witness camera’d the actress and the moving lights so we could see exactly where the light was at any time. Then we replaced her head from the neck up with a CG face and hair, modeling minute details into the digital head. We scanned Loren and Sean at ICT [the University of Southern California’s Institute for Creative Technologies]. And we found a life cast of Sean when she was 28. She was 20 when she played Rachael. So beautiful.

We added some blemishes and flyaway hairs in the hair groom to make the digital Rachael look more real. We worked on the model to get a CG face that was indistinguishable from Sean Young in the original Blade Runner. Then we showed it to Denis and the producers. The response was that this looks like a girl that looks like Sean, but it isn’t Sean. So, we focused on Sean’s eye specifics, added 80’s make-up, and did a bee-sting design for her lips like a 40’s femme fatale. After we made those adjustments, we inserted our digital double into one shot in each of three original scenes from Blade Runner and showed them to Denis and the producers. They asked why they were looking at the old movie. I told them that one of the shots they had just seen in each sequence was the digital double and they were blown away.

MPC’s VFX breakdown from Blade Runner 2049, including Rachael reborn.
MPC/YouTube

I wanted our Rachael to walk up toward Deckard with the confidence the original Rachael had then, when she gets there, move into longing. And then he rejects her. We found all those emotions, confidence, longing, rejection, in Sean’s original Rachael and matched those tells, those subtle nuances.

Denis got to direct the performance three times: Once with Loren on the shoot day with Harrison, once in the facial motion-capture session where we captured Sean Young and Loren Peta performing, and the third time with us and MPC in post, where he could comment on the animation moments as Rachael reborn acted her part.

What about the other digital characters?

We had so many when you think about it. Elvis — bringing back Elvis as a hologram in Vegas. Frank Sinatra. Marilyn Monroe. We created them in a similar way we created Rachael and Joi. We hired impersonators. The day we shot Elvis, everyone was singing Elvis tunes, but he didn’t really look like Elvis. Nor did the guy playing Sinatra — he was heavier and bigger. But they had the mannerisms down. The woman who played Marilyn looked quite a bit like her.

Did you pull out any other tricks to make this film?

I spend endless amount of Saturdays working on the merge and Rachael reborn and then had the other work on top of it, so I pulled in Habib Zargarpour as part of the production to help me in post for eight weeks. We were viewing so many shots — over 100 shots a day. It was overwhelming. I referred to him as the effects special forces. I’d have him parachute into a place that needed work.

What was the best thing about working on this film?

I got to do a film I had a great deal of respect for, but that was just the beginning. The best was working with Denis. He’s such a good director. He knows what he wants and goes out and does it. He makes decisions. Leads by example. And above that, he’s brilliant and wonderfully nice. I also had the honor of working with Roger Deakins and Joe Walker, who were both great and helped VFX quite a bit. Karen Murphy is a great visual effects producer and kept us all together and on track. And the imagery is exactly the type of effects I like doing. Subtle, emotional. Not like the constant din of someone banging on a pan.

The merge is what I’m most proud of, but I’m proud of every scene. For this movie, the visual effects supplied the scale and scope of the film and then on the other side, we got to do character work, the subtle stuff. It was like we were making magic. We had to make up the definitions and make them right. I love cinema and I love solving problems so this was a double whammy of that in spades.

And I got to make some great imagery, which is always a real joy, and balance it to work in the fabric of the film to give the director the vision he desires.

The hall of replicants
Alcon Entertainment

What was the hardest?

The cities, the merge, and Rachael. The most difficult was the merge. But the cities were hard to get right and Rachael, too. The hall of replicants – to make those people barely move.

What did you learn?

Instead of having lots of short shots, single long shots can be as effective and maybe more so. Great imagery can be brought together by balancing things. I did a lot of mixing effects takes.

What do you mean by “mixing effects takes?”

Say I have version 10 of the effects and two weeks later I have version 50 and two weeks after that I have version 80. I’ll take something from version 10 and lay it in, then lay in version 50 at a top level, and then 80. I might like this little part from 10, so I’d bring things up and down almost in real time. I know it’s crazy to do that on every composite, but I did a lot of it. It can be really effective. I’d take what I liked and turn it around quickly. It gave the film a great analog feel. Things felt more human. It worked particularly well in the memory lab.

What are you working on now?

I told my agent I was taking six months off. But I hadn’t planned on awards season. My wife said, “When do the six months start?” So now I’m going to take a couple months off. I’ll keep my nose down, walk around the house, and see what needs to be fixed. And then I’ll see what movies are out there. This will be a hard act to follow.