“He was saying, ‘Oh, it’s just two people kissing,’ Zapara recalls. “I’m like, ‘OK.’ He says, ‘It’s very close up,’ and I’m like, ‘That’s fine.’ And then he says ‘Over the course of the video, they age.’ And I said, ‘What?'”

Aging effects are nothing new, of course, and most VFX jobs would simply employ multiple stages of make-up effects and a series of quick morphs to get through the process. But Klasfeld wanted his two performers to age ever-so-slowly in a single shot over the course of a three-minute music video, which turned out to be a fairly tall order. Zapara and his team at HAL – a VFX firm founded earlier this year by Zapara and Klasfeld along with director Tony Petrossian and original VFX producer Ed Irastorza – ended up creating an elegant Frankenstein’s monster of a video, patching together bits and pieces from multiple takes (including performances by two different sets of actors) into each frame to create a convincing, almost imperceptible vision of the aging process. Watch the video, then read the Q&A below to find out how they did it.

Our visual-effects producer at the time, Ed Irastorza, sat down with me, and we figured out how we would try to pull this off. We’ve seen aging effects before, but usually never more than five seconds at a time. This was going to be two and a half minutes or so of continual aging. We were thinking – or hoping – we could get away with shooting two actors in various stages of make-up, choreographing them as tightly as we could to do the same thing over and over, and then morph them at five- or 10-second intervals. At a certain point, because Marc was worried that the old-age makeup wouldn’t be convincing, we decided to find an older actor and actress to play the young actor and actress. We did some reverse makeup on them, and then aged them through their deaths. And they were choreographed the same way as the younger actors.

Now, when two people are redoing the same take over and over, depending on how good a mime you are you can get fairly close but it’s never going to line up. Going between different people adds a whole level of complexity to it. Your mannerisms are going to be different from the other person’s, even if you’re trying to mimic them. We found that out over the course of the production. We shot several takes of each with several seconds of overlap between them – usually we went back a full verse of the song, so we had a lot of overlap between all these takes. We had thousands and thousands of frames. And then we went through in the editing process and picked the different selects of each take. “Oh, I like her eyes in this shot. I like his face here.”

The final frames are almost all metaframes ‘ a combination of several frames. Very few frames in this video are from just one take, and almost all of those are in the beginning. Pretty much everything else you see doesn’t exist as any one take. It’s a combination of 30 percent of this face, 20 percent of this part of that face, all put together and re-tracked, re-roto’d and re-morphed into the other bits.

What system were you working in to composite the shots together?

The majority of the warping and the tracking, the lion’s share of the hard work was done in Shake. The majority of that work was done by two Shake artists, Bill and Zoe Eyler.

Did it only become clear once you were in compositing that’s how you were going to have to build the effect?

Yes and no. We knew from the beginning that Marc wanted to be very subtle. This wasn’t going to be Michael Jackson’s “Black and White” video. As cool as that was, ours wasn’t going to be these two-second morphs. So we said, “Let’s push this to 10 seconds.” So we have 300-frame transitions, which is pretty big. And then we’ll have 300 frames in between. And then another 300 frames of a transition. We still had a significant amount of work to do, but we had what we called points of reset – a place we can always go back to. As we were working, it turned out that the subtlety we needed really overlapped into everything. There was almost no place where we had a reset. Because everything was an effect, there was no place to hide, especially because of how close we were. You could see every nook and cranny of their makeup, so an age line in one place had to line up with the age line from the other takes. Any little mismatch in the stabilization and the reverse tracking really showed. There was quite a bit of shuffling work to do. Since everything was moving, there was really no reference point. You were tracking against tracking for minutes at a time. It turned into a very complicated task.

What format was it shot on?

It was shot in 35mm, and we did a high-def transfer to 1080p. We always knew we would just have a D1 master, but we shot it wider than we expected. That helped quite a bit. Since we knew we would have to do significant tracking, we had all the tracking information off screen so stuff wouldn’t get lost on the edges. Even if it was off screen, the tracking markers would remain. Right after the editing process, we found out where Marc wanted the final framing to be. If you get in too close, the composition is too graphic. There’s no acting involved once you’re that close. So we found a nice medium between that graphic look and getting all the nuances of the acting, and then used that as a general mask as the artists were working on the comp. They knew, “OK, we’re tracking this head, and the hair’s really difficult, but at this point in the frame we won’t see it anymore, so we let it go.” There’s a lot of slop going on on the master frame.

How did you manage the transitions between the two sets of performers?

That transition ‘ between Lucas and Annie, who are the younger folks in the video, and Stu and MaryLou, who are the older folks in the video ‘ is, I think, the hardest part of the video. Not just because you have different faces ‘ as much as we tried to make them look the same, their noses are in different relationships to their eyes and their mouths. Not only did that have to be subtly done so they didn’t look rubbery, but the ways they moved are vastly different. On a certain note in the song, one guy may have his head down, and the other guy has his head up. When you’re doing take after take after take, you don’t notice those things on set. We knew there would be some discrepancies, but there were large discrepancies. And that’s where having the full-frame face really helped. That all has to be completely reanimated, and information on the edges is completely lost. Thankfully, we were in close enough that it didn’t become a problem too often.

And each time you did the shoot there were several stages of actual physical makeup?

Yes. This was a video with a very low budget, so we couldn’t go for the full cyberscans of the actors. We didn’t have 3D to work with, and we couldn’t do complete face-mold prosthetics. We had to re-use appliances, which are more generic and they don’t quite fit as well. And this was supposed to be a one-day shoot, but we had to make it a two-day shoot simply because of the time to go through appliance make-up. We had to temper our make-up needs – just how much makeup could we get done in the shooting schedule, and how many takes could we do? We were going rapidly between these people, and we had to cut a lot of corners on make-up because we just didn’t have the time.

Were you shooting them both at the same time?

We shot them both at the same time, and then we shot what we called the “singles.” Usually one actor went into make-up, and then we had a stand-in just off-frame so they had someone to look at.

And that allowed you to get more elements to use in the final piece.

Yes. There wasn’t as much roto to go over. It was shot on green-screen, but once they start interacting it turns into roto. Not that there wasn’t a ton of roto to do.

It seems like the shots where they’re making contact ‘ the kisses ‘ would have to be largely from the same take.

We choreographed so that for the stuff that we knew was going to be the hardest, we didn’t have them kissing. If you watch the video several times, you’ll notice that while they’re kissing the majority of the changes are going on behind their noses. I was worried mostly about the hair. I always notice that in morphs the hair is always the thing that doesn’t look as convincing. And they did a fantastic job, not just matching the hair in make-up, but getting that hair to line up and turn grey over a very long period of time. In fact, on the first version of that effect, we were watching and looking at the face, and we were like, “When do you think you can start the hair?” It was so subtle that we were even fooled the first time around.

I was trying to scrutinize the video to see exactly how you did the transitions. And I realized that while I was looking at one thing on screen, trying to see what was happening to it, everything else had changed without me noticing it.

That was the goal. I always say VFX is an exercise in misdirection, and knowing where to pick your battles. But since we had no place to hide here, we had to make everything very subtle. To find the flaws you had to look very closely in one place. And while you did that you missed everything else. That was a huge, huge task, and I think the guys who pulled it off did a wonderful job on it. If you look at it from a VFX standpoint and you realize what had to be done to generate this at the length it’s at, and with the close-up fidelity it has, it’s hard to think of something comparable. I’ve seen aging effects done grander, done better. But if you put all those things together and do it over the course of a two and a half minute piece, well, there’s a reason why that hadn’t been done before. As we all found out!

Other than Shake, what tools were used?

There was a lot of what I call rig-removal. The actors were wearing T-shirts, and Marc didn’t like the way the shirts messed with the composition when they came into the shot. A lot of that removal was done in After Effects. At the very end of the shot, on that dolly out when we see that they’re on the street, every place they’re walking there was a dolly track. That was 3D-tracked in Synth Eyes and removed in LightWave and After Effects. All the band members are roto’d back into the plate. There was a lot of back-and-forth in R&D on the look as they age and fall apart. First it was supposed to look like porcelain, and then it was supposed to look like dust. We went through a lot of variations. Zoe and Bill were going to do that part, too, but we realized there wasn’t going to be time, so we transferred it to another Shake artist, Steve Wright, and Steve and I worked on it. Again, we didn’t have the cyberscans to do a full 3D simulation. We did use some 3D dust, and I hacked together a kind of front-projection of 3D shells of faces based on generic heads in LightWave, which I composited in After Effects. A lot of the dust was done in Lightwave and composited in After Effects with real elements, and some CG elements were done in ParticleIllusion.

Frankly, we waffled on what that look was supposed to be. The hardest part in that portion of the video was just getting something Marc was seeing in his mind’s eye. We had a hard time getting the subtleties of that bit, and there was a lot of back and forth on the smoke and the dust. As with many visual effects, we took a long, winding path to get to a nice end product – but we did eventually get there.

Credits:

VFX Supervisor: Chris Zapara
Shake Artists: Bill Eyler, Zoe Eyler, Steve Wright
After Effects Artists: Chris Zapara, Judith Holzman
Makeup FX: John Goodwin
Makeup FX Assistant: Brian Kinney
Producers: Ed Irastorza, Bill Newcomb
Actors: Lucas Field , Annie Kates, Stu Levin, Mary Lou Secor