It figures that the guy who invented Photoshop (with his brother, Thomas) would figure out a unique way to create a CG character. And so John Knoll, the VFX whiz who’s been inventing new ways to get a special effect throughout his professional career (see Barbara Robertson’s profile of Knoll, published in Film & Video last year), saw Pirates of the Caribbean: Dead Man’s Chest as an opportunity to implement a new process – dubbed "Imocap" – for grabbing motion-capture data during principal photography. The result is a crew of waterlogged CG villains quite unlike any other. We asked him about the new process, its transparency to the rest of the film crew, and how it might be used in the future.

F&V: Tell us about the "Imocap" process.

John Knoll: Hal Hickle, the animation supervisor here at ILM, and I got together at the beginning. We had the situation on Pirates where the characters began as fully costumed actors and in some shots they would transition into CG skeletons and back. So it really made sense to photograph them in costumes and track movement manually and transition back and forth to CG characters. In Pirates 2, because the characters are always CG and don’t have to transition back and forth, there was an opportunity to try and do better than a manual matching process. We had done some mocap with Johnny Depp on Pirates 1 for some of the Skeletal Jack moments. But it was different because Gore wasn’t directing there. Hal and I flew down and did a mocap setup in L.A., but it wasn’t the same. You weren’t on set, you didn’t have the other actors to act against. And without Gore there to direct the nuances of the performance, it became a difficult thing. We only used two or three shots from mocap, and the rest was done by manually matching the film performance. Gore had cut with those film performances and really wanted to match them exactly.

We started thinking, “How can we get better data then we had on Pirates 1?” – which was, really, nothing. Imagine a two-dimensional graph where one dimension, the vertical axis, is the footprint on set and the other axis is quality of data returned. Pirates 1 was the lower left-hand corner – we had a low footprint on set, we didn’t bring motion-capture gear and try to get that on set. But we didn’t get good data. It had to be painstakingly, manually tracked and matched. The upper right-hand corner would be a full-on motion-capture setup. For that, there are constraints on lighting and camera setup and what can obscure the view, a whole bunch of machinery, and an enormous footprint on set. I couldn’t take that on location – imagine trying to use that in knee-deep water on Exuma. We tried to work with the R&D department and aim for the lower-right-hand corner – a rugged, robust system that can take some abuse on location, places where there’s no electric power and extreme conditions, and not put any additional constraints on the filming. It wasn’t going to slow anybody down. When it’s costing you that much money per minute, you don’t want the first-unit crew to have to wait for you for five minutes while you do this or that.

What we wanted to get out of it was, essentially, motion-capture data. And the R&D team came up with this Imocap process. You put the performer in a suit that has tracking marks on it – a different sort of tracking mark – and then you have two high-resolution witness cameras that you bracket the film camera with for triangulating positions, and that’s really about it. The rest is clever software that R&D put together to reconstruct skeletal motion.

You said one of the reasons you came up with this process was that you had some frustration with the data you got from film performances on the first Pirates …

Well, because of the nature of that show, we couldn’t really have any substantial tracking marks to aid that process. It was all done manually, and there’s a lot of interpretation done as part of that manual fitting. You get most of the way there but there’s some subtlety that you don’t get. That’s where the motion-capture quality comes in, but it needs to low footprint and portable. The other thing that was very important was that it all be able to happen on set. You don’t want to set up a situation where your characters are not really on set. When Davy Jones and Jack Sparrow are having a dialogue scene, they really have to be there on set so the dynamic between the actors is the same as live action. The director and DP have to have someone to direct, someone to light. And then the editor has a performance to cut with. We’re going to do it all right there, on set, like any other live action. There’s no stylistic break, no place where, suddenly, the CG has a different feel from the rest of the live action, no. It feels exactly the same way.

Did you offer guidance to the cinematographer in terms of how the lighting would affect the CG characters?

I worked with Dariusz [Wolski] before, on Pirates 1, and we have a very good friendship and professional working relationship. He’s been very friendly and helpful to our goals. I, hopefully, have been respectful of his intent in lighting the movie. I tried to make it clear on Pirates 1 that, when we have the gray and chrome sphere that we bring out on set to measure the on-set lighting, our intent is to duplicate the lighting as faithfully as possible, at least on first pass. And then I do the same things he would do looking at an actor under lights. Take the fill down a half stop and bring up the backlight. I depart from the on-set lighting to make things look better.

It’s hard to light a character that isn’t there. He lights Bill Nighy. The reality is later we replace it with a CG character and the quality of the scan is different, how it responds to backlight and soft bounces is just a little different. You make adjustments to make it look better.

How about performance? Did you talk at all about how broad or subtle the actor’s performance should be to come across through the CG character?

No, really, that was all between Gore and Bill. They discussed how broad the character should be played, and Bill was without a whole lot of information – he’d seen the designs of the character and knew where we thought we were going with it, but he didn’t see any final results. It was a leap of faith on his part – I’m going to play it this way and trust we won’t make it look silly later. I thought he did a fantastic job. He also gave Gore a ton of options. He’d do five takes and every one would be different in some kind of wonderful way – a little quirkier, a little ironic, a little more anger and bite on the end. There were a lot of interesting variations.

How was the animation process, in terms of making sure the director was able to guide and approve what the FX team was doing? Was it a departure from other films you’ve done?

It was fairly similar to other projects I’d done, except in every shot we had these wonderful filmed performance of all the CG characters. There was no question about how exactly to play a character. These were decisions that were made on set, and we were perfectly fine with them. One thing Gore learned in cutting was that we could bridge performances in one shot – he could use two different takes and cut two different line reads together with a hard cut, knowing we could smooth it out and make one nice continuous performance. He gets the best of both worlds. We also had the opportunity to fix little continuity things. Bill did a head-turn in one take and he didn’t in the other and Gore wanted to cut before the head-turn happened and repair the continuity. The performance was under our control, but the basic procedure was to match Bill’s performance as faithfully as possible.

And the rest of [Nighy's] crew were all very good actors. I had a booklet of artwork and designs, and to a one all the actors were interested in studying the artwork and making choices based on that. They were very enthusiastic about that, and that’s something I just loved.

Now that you’ve developed this technique, do you expect it to be standard procedure for ILM? Or does it depend on the project?

This clearly works well when the character is human-sized and there’s something to be gained by doing the capture there. I think it worked very well in this case because these characters started off as people. They’re human-sized and they follow a human form with two arms and two legs for the most part. Being able to do it on the set had all the advantages I talked about earlier. The direct interaction with director, camera operators, and between the CG character and the rest of the live characters, and having that close interaction on-set. It’s a requirement for good performances, really. Any time you can do that sort of thing, you’ll see a set-up like this because I think it offers huge advantages. It will not work particularly well if a CG character is some really odd alien form that you can’t capture any reasonable motion from ‘ he has eight legs or he’s some odd proportion or size. I don’t think this would work very well for a character that’s 20 feet tall or six inches tall.

Did you use this system to capture more detailed information specifically for facial expressions?

No. Facial performance is an ongoing R&D project but we didn’t do it here.

Are you already making progress on Pirates 3?

Well, fortunately, Davy and his crew are back. A lot of the challenge of Pirates 2 was getting that process working to the point where we could crank out shots efficiently. We got there around March and are going in [to the new film] with that part of the pipeline working smoothly. But there’s a whole lot of new things we’ve not ever dealt with. It won’t be an easy thing.

>> Read the interview with Dariusz Wolski.