HDV and Old School 3D

The newly re-opened Hall of Human Origins at the American Museum of Natural History in New York City tells the story of human evolution from our earliest ancestors through the development of modern language, art, tools and technology. Outfitted with the latest techniques in exhibit interactivity, it’s a sight- and experience- to behold. But nestled among all of the Hall’s 21st-Century features is the La Micoque diorama, an updated icon from the original hall that uses technology from another century.

The diorama depicts the famous archeological site of La Micoque in Southern France where excavations in the late 19th Century helped to provide insight into the day-to-day life of ancient humans 300,000 years ago. Set within the cliff-site diorama are a miniature scaffold, table and shed into which miniaturized live-action scientists working on the site are projected using what’s called the "Pepper’s Ghost" effect.

Pepper’s Ghost is an illusionary technique that uses special glass and lighting to create the appearance of an object projected into a three dimensional area. This effect was originally used in theaters and magic tricks to make objects appear, disappear and morph into one another in front of people’s eyes. Think Princess Leia’s hologram projected by R2D2 in Star Wars. Developed originally by inventor Henry Dircks in 1862, the "Dircksian Phantasmagoria," as it was initially known, was made famous by John Pepper’s implementation of it in a scene from Charles Dickens’ The Haunted Man at the Royal Polytechnic in London that same year.

The effect relies on a special half-silvered mirror positioned at a 45-degree angle between the viewer and scene, a controlled lighting environment, a special viewing window and a hidden area where the object to be projected is located- all straightforward to implement in a diorama.

The original La Micoque diorama, which debuted at the Museum in 1993, employed the use of small television monitors hidden in front of and below the diorama, which ran a video loop of the scientistsworking. The images from the monitors were reflected onto a special mirror to create the effect. As part of the refurbishment of the Hall, Mindy Weisberger, the Museum’s assistant director of video production, Exhibition Department, brought my production company, 24fps Productions, in to help update the video for the diorama. "We wanted to preserve a sense of continuity with the past, and the Pepper’s Ghost effect was tremendously popular in the original hall. In spite of the simplicity of the method," and its 19th-Century origin, says Weisberger, "La Micoque consistently had visitors wondering,‘How’d they do that?’ It is still a highly effective means for creating a magical environment in which moving figures bring a scale model to life."

To give the effect a modern gloss, the Museum would be replacing both the diorama and the media "projected" inside it. We had worked with the Museum in 2001 on the popular "Pearls" exhibit, where we created what was perhaps the first underwater HD chroma key shoot. This project also required us to shoot actors against a greenscreen. Though it didn’t involve the plunking of cameras and large greenscreens into the water, it did have its own challenges. There would be three separate scenes that mimicked archeological work out in the field: an archeologist kneeling and excavating (which would be imaged onto the scaffolding of the diorama), a palynologist working at a table (placed next to a miniature model of a table), and a zooarchaeologist working at a table inside the shed. We would then key the green in the scenes to black, and finally give it to Lee Patrick, the Museum’s media producer, for final processing. We decided to shoot the video in HDV with Sony’s HVR-Z1U camcorder. Although the new LCD monitors for the diorama weren’t high definition, the extra pixels of HD let us forgo setting-up ratio calculations to match the size of the actors with the size of the model. Lee could simply resize the video to make it exactly match the scale he needed.

Pulling a good key was one of the most important factors in this project, so we devised a series of tests to compare different lighting setups, clothing colors and textures, props, keying software, colors and our overall workflow. We compared each of these variables against one another, ending up with a ton of useful data.

Our main lighting issues were making sure we had enough separation from the subject and background key, and preventing green spill onto the subjects themselves. We determined that a better key could be pulled if the subject were closer to the camera- up to a point. We found that the best key could be pulled using two front lights, each at a 45-degree angle from the subject. We used an ARRI 1K with Chimera Softbox for the key light, and an ARRI 650 with diffusion, mounted on a swing-arm, for the fill. This way, we could easily tweak its position for each scene. The background key material was lit with four 500-watt florescent color-balanced light banks, custom-built by my associate, Michael Frenchman. These lights also helped give us the right amount of hair lighting needed for good separation.

Our post workflow involved ingesting the HDV into our Final Cut Pro HD edit system and converting it to 1920 x 1080 8-bit HDCAM. Since HDV is an MPEG-2 format, it can be problematic during the edit; we’ve learned to always initially convert the HDV source into another format. After editing down the footage, we keyed our selected clips to black, stored the exported HDCAM files on a hard drive and gave them to Lee so he could scale and integrate them into the exhibit.

We had initially considered using Serious Magic’s Ultra 2 software (now wholly owned by Adobe and sold as Adobe Ultra 2) for the keying. But since Ultra 2 wasn’t available for the Macintosh platform, and the work-around with such large HDCAM files would have been problematic, we decided to go with dvGarage’s dvMatte Pro, a keying plug-in that works directly within FCP as a filter.

As one would expect with a greenscreen chroma key keyed to black, clothing with shades of green or dark colors didn’t work very well. We also found light khaki shirts and pants to be too reflective, picking up too much green spill (there’s a reason they are a top choice with archeologists who must dig all day in the hot sun). Light blues, darker khakis and grey worked better and absorbed most of the spill.

Props that had thin or spindly features didn’t work well- they would disappear into the background once the key was pulled. Ziplock bags, which we used to simulate the collection of artifacts, worked very well. Their translucent features keyed evenly with the background, even while they were being moved around the scene.

We also tested and perfected blocking strategies for the actors, so the looping of the footage with props would be more seamless. Finally, we determined that white balancing against a WarmCards "Warm 2" provided a nice color tone that would hold up well with the Pepper’s Ghost effect, which can, as its name implies, sometimes render the images too faded and ghost-like.

Having done such an exhaustive test, we found that the actual shoot went very smoothly. The Museum supplied the actors- grad students in archeological programs- and we shot several takes of each of the three scenes. To maintain the largest possible pixel resolution, we positioned the camera vertically, using a 90-degree ball mount for two of the scenes and turning our monitor on its side to view the scene correctly. We also draped extra key material on any surfaces that weren’t going to be in the scene; this saved us lots of time when pulling the key.

After we ingested the footage into FCP, picked out the best takes and converted them into HDCAM, we set about the task of pulling the key. This was easier said than done. Despite all our tests, we realized that no lighting setup is perfect, nor is any keying software. We needed to develop a workflow that would clean up these imperfections.

The first step was to crop all unnecessary areas out of the scene. We then applied an overall key with the dvMatte Pro plug-in filter, studying the result on a high-definition monitor. Rarely did this single key work on all areas of the scene. Just as we were getting one area to pull well, the actors’ hands would start to disappear. Adding additional key filters to the same clip would only exaggerate these problems. To address this, we duplicated the entire clip for each new key we needed, stacking them on top of one another in the timeline. Then we used garbage mattes to isolate the desired area of the scene requiring its own key, and we applied a separate dvMatte Pro key to each. In order for the non-matted area to be hidden (so black doesn’t show through), each higher-stacked clip needed not only its own matte, but also an inverse matte of the lower clip. With multiple duplicate clips stacked on top of one another, it was a lot to juggle.

Once we applied all the necessary keys and mattes, we rendered each three-minute scene, which took, on average, eight hours per scene. Occasionally, we encountered problem areas that became visible only after rendering, adding time to the process for correcting and re-rendering. The entire project took a couple of weeks to complete.

The final scenes still had minor issues, especially when viewed on our high-definition monitors, but we determined that once the material was downconverted to standard definition, these problems would subside. Although a 2/3-inch chip camera may have given us a better signal with which to pull the key, the choice to use HDV worked out well.

As Lee was looping and scaling the scenes we had delivered to him on hard drive, experts from the Museum were modifying the original diorama to fit its new home in the newly renovated Hall of Human Origins. Alec Madoff, the original model-builder and the Exhibition Department’s senior principal preparator, cut down the model to fit into a smaller space, while scenic artist Jack Cesareo painted a beautiful new backdrop. A digital video server fed the scenes to monitors installed inside the completed diorama.

The Hall re-opened to the public on February 10, 2007. If you get the chance to visit the American Museum of Natural History in New York City, make sure to check out La Micoque. We were honored to be a part of a project that has a permanent residence at the Museum and even more thrilled to show how an old visual trick, finessed with the latest technology, still has the power to enchant. n