Realizing an Epic Final Battle on the Streets of New York — and a Smashing Hulk — for The Avengers

All the heroes for The Avengers don’t appear onscreen. With a film this dependent on visual effects, you’d have to consider work by the crews at all the studios who completed the highly-praised shots on a tight schedule as heroic. Of those studios, ILM artists created 700 shots, and supervised water work by Scanline VFX for a sequence in which a “helicarrier” built by ILM takes off from the water.

Early in the film, the superhero Avengers sort out their differences in superhuman fights. ILM’s Iron Man has a lot of action around the Stark Tower, which ILM’s digimatte artists built, and the ILM crew found imaginative ways to put Iron Man into and out of his suit. The studio also had shots of Thor fighting a CG Hulk that critics praise as the best Hulk yet.

But ILM’s biggest shots happen during the last third of the film, when the Avengers band together, gather in New York, and turn The Hulk loose to battle Loki and the aliens. Principal photography for this battle, however, happened in Cleveland, Ohio and on a greenscreen set in Albuquerque, New Mexico where the filmmakers installed a 300-foot replica of the Park Avenue Viaduct. ILM created the visual effects for the 30-minute climax. "We extended that set digitally on all sides," says Jeff White, visual effects supervisor at ILM.  "And, we did most of the work in the big battle."

To create a digital New York, ILM sent a team that photographed all the locations in which the battle would happen; previs created at The Third Floor showed them the vantage points they’d need to create. "We had Canon 1Ds Mark III cameras attached to automated heads to shoot 360-degree photos with overlap,” White says. "We were all over the city on rooftops, hanging cameras off buildings – you can’t fly low to get aerial plates in New York. We shot on the ground every 100 feet and up every 120 feet with a man in a lift moving down Park Avenue. We stitched the images together to get high resolution photographs of the environment, and in all, we probably made 1800 spheres, each with 72, 12K pictures. It was really amazing. It was a lot like a Google street view but in high resolution.  I think we had 275,000 photos of New York.”

 The film crew in New York also shot reference plates of the buildings ILM would recreate and a few key areas. “We wanted to get the feeling of how New York looks on the [ARRI] Alexa camera, and EFILM pre-timed a lot of that material,” White says. “We had good reference.”

To build the digital city, modelers at ILM created the buildings that would appear during the battle by working from Lidar scan data that Marvel had captured.  "The Lidar scans helped us create great high-resolution geometry," White says. "Our layout group brought in the geometry and the spheres we had recorded, and did a solve to line them up.”  

The next step was projecting the images onto the geometry. “We pulled images from the spheres and blasted textures onto geometry using a splat technique,” White says. It’s a technique often implemented by game developers to blend high-resolution textures using alpha maps.

“We get the best fit from wherever the camera is,” White says. “When we're flying the camera down the street, the splat shader will take all the spheres along the way into account, paste them on the buildings, and do a rough occlusion. Then, we could fly [a virtual] camera through the areas of [digital] New York so Joss [Whedon] and Janek [Sirrs] could make decisions about the backgrounds.”

With backgrounds selected, the ILM team began the laborious work of fixing the environment. “The photography gets you 70 percent of the way there, and it gets you there quickly, but the real work was putting in all the stuff lost in photography,” White says. “Imagine you’re in a man lift photographing Park Avenue with all the beautiful trees, cars, and things on the street. Some of our shots were nodal moves – panning around the scene – and we could almost use the environments as is. The buildings tended to work because we had enough material to do multiple projections and blend between them. But, when we needed to fly through the scene and change perspective, we had to replace the trees, cop cars, taxicabs, sandwich boards, awnings on buildings, hot dog stands – everything you’d see on a street corner – to put parallax back into the scene.”

The team also replaced all the windows in the buildings using special RenderMan shaders to insert correct reflections, random interiors and window blinds. They added building sheens, Fresnel reflections, sun kicks off the windows and other lighting effects, waving flags, smoke, and vapor from air conditioners, digital doubles walking along the street, even construction cones. White estimates that the crew created 190 models for street dressing, many animated.

“Photography is static, but when you look at a city, there’s always some movement,” he says. “And with any city assembly, variety is the key. Our digimatte artists, lighting artists, technical directors, and set dressers worked hand in hand to dress the photography. You can imagine the data organization challenge. We started building New York in June as soon as we got the photography back, and we were working on it right to the end.”

Once the crew had built an environment, say the Park Avenue Viaduct that leads to Grand Central Terminal, with all its 3D geometry, windows in the buildings, cars on the street, and so forth, they considered the whole as an “uber” asset – an asset in this case that CG supervisor David Meny and digimatte artist Andy Proctor built. “We could put the camera anywhere in that environment and get a good result with our ‘box render’ almost automatically, including the Fresnel passes on the buildings,” White says.

Sometimes the lighting in the original photographs worked, but often the lighting artists would add shadows and light kicks through the buildings to help fit the CG characters into the environment more realistically.

Inside the environment, ILM inserted a race of aliens, digital doubles for the Avengers, a CG Iron Man in his new Mark VII suit, and a CG Hulk. The Hulk was the hardest.

To create the angry green giant, the team started by making a digital double of Mark Ruffalo, the actor who plays The Hulk in his human form, using data captured from Mova sessions, photographs, life casts, and from USC ICT’s Light Stage system. Then, they buffed up the digital double to create The Hulk, using new volume-preserving rigs. For his performance, the animators worked from data captured from Ruffalo. Some of the effects in the end battle shots were practical, and New Deal created some with miniatures. For others, ILM added CG pyro using the studio’s Plume software. And, of course, the CG characters and digital doubles.

“We hope that people will watch the battle and enjoy The Hulk and other characters, and not think about the environments,” White says. “We were excited about these shots from the beginning. The fans wanted The Hulk to cut loose and that’s what the director let him do. We wanted to get to the point through the photography and the process we went through to make environments feel as close as they could to background plates with The Hulk jumping and smashing, people running away, cars flipped over, and the Avengers surrounded by aliens. That’s when we feel like what we’ve done is working.”