The booming gaming market, films like Polar Express and increasing use of animation in commercials has challenged animation studios to create near-photorealistic 3D CG-animated characters. The problem, however, is that while today’s animation software allows the creation of such characters, the task of making them move and react in a realistic manner is arduous to say the least.

Many have turned to motion-capture to streamline this process but the mo-cap on facial movements translate well to digital characters, making movements stiff and stuttered, certainly not conveying the emotion you want from an ‘actor.’ So animators are asked to create this performance manually. Even if it is achievable, the process is so time-consuming most projects simply can’t afford it.

This is the conundrum that led Softimage to create FACE ROBOT – at a price tag over $100,000. More recently animation and digital FX studio Pendulum decided to devote its R&D team to developing its own system and came up with StretchMark, a motion-capture analysis and implementation tool that works in within Autodesk Maya to create realistic facial animation from data gathered from the VICON mo-cap system.

Click below to watch the test animation Pendulum created using StretchMark using a monologue from Shakespeare's 'Antony and Cleopatra.'

…then Watch the side-by-side comparison of the live action footage from the motion capture stage with the facial animation below…

Why did you decide to invest the time and money to create an alternative solution?

Taylor: We felt like there wasn’t any methodology out there that was achieving successful facial animation cost effectively, or even efficiently. We’d seen some approaches that looked just okay. We didn’t feel like Polar Express was achieving what was possible, we felt there was a creepiness factor to the characters. It was somewhere between realistic and animated, just enough to look strange.”
Their development goals were twofold: efficiency and quality.

We felt there were methods that could be implemented that would be better. Essentially what we set out to do is allow us to capture tons of mo-cap facial animation, high data levels and apply it really fast to give us an efficient pipeline for the creation of facial animation with respect to motion capture data that we are getting back from the VICON system.

How does the StretchMark work?

McCormick: For each of the control points on our digital head the software defines an area of influence and tells each of those points to look through the database of blend shapes and affect its area by combining any number of the blend shapes that it wants so it can get as close as possible to the original positional information of the motion capture data. Because it is not going through any bio-mechnical analysis or mathematical analysis the mathematics are very clean and very fast and essentially instantaneous. We apply the data – we call it data-fitting – with StretchMark to the face and hit APPLY and it’s done. It doesn’t have to go through a baking process. We were pretty surprised at how solid that first pass is. After that first pass you have better facial animation on your characters than most of the stuff we’re seeing coming out of anywhere. From that point we realized there are going to be little breaks, points where the smile is too sharp and stuff like that. So we go through a phase we call corrective blend shape identification and that allow us to identify points that need a little sculpting tweak and we create X number of corrective blend shapes to add to the pool of blend shapes that StretchMark can call upon. Then we can fix any of those vertex violations and run the pass again and now it uses instead of just the 20-30 blend shapes it uses those plus another 15-20 corrective blend shapes and you’ve got pretty much a final pass on the animation that’s in good shape.

Does this data-fitting and corrective pass have to be done scene by scene?

Taylor: No. Once the motion capture is applied it is off and running so that once it is applied it is applied for the entire character. If you have someone talking for a half-hour on screen and it takes an hour to data-fit that is all the work required for that half hour of that character. If your motion capture is good enough and you applied the right blend shapes you can apply hours of facial capture all at once.

What if you want to change the movement of the digital character to make it different from what the actor on the mo-cap stage did? Are you back to manually animating each movement?

Taylor: If you want to change the performance ‘ say he blinks too much or doesn’t smile enough – we even have it so you can apply non-destructive animation controls on top of that.

We can also set up multiplier to exaggerate movement. We set up a couple different control points like a cheek point and can hand it off to an animator, who knows nothing about StretchMark, and he can puppeteer the face to do anything they want on top of the data and those changes are applied in the data of StretchMark

And the digital character can be anything, not just a representation of the actor captured?

McCormick: That’s one of the benefits of StretchMark, it can be a cartoon, we can apply the facial animation to an animal, any character you can create. We reapplied the Mark Antony monologue to a cartoon looking monkey just as a test.

Did you develop this for a specific type of project?

McCormick: We’ve been doing so much on the game cinematics and commercials that we needed something that would cross both genres. On the commercial side we needed something that would allow us to create something very realistic. On the game side we needed something that we could apply tons and tons of data to for a 4-minute cinematic or even be able to go to a game client that has 10 characters with 40 minutes of facial animation for each character in the game. They will send us their characters, we’ll develop the blend shapes based on what their game engine can handle and apply the facial data to their characters and hand it back to them. The amount of mo-cap we can apply to them is massive. Since you do that work on the front end and if that work is good and the blend shapes are good you can apply data till the cows come home and it is extremely efficient. Because of that the gaming work will probably see the most significant improvements from an efficiency standpoint.

Taylor: Because we wanted this for both commercial and games [for our test] we picked something that would challenge the emotional acting aspect we were trying to achieve. We leaned on Shakespeare with the Mark Antony monologue to show we could make a digital character emote.