Siegfried Foessel

Siegfried Foessel, Fraunhofer IIS, on Getting the Most Flexibility in Post

As head of the Moving Picture Technologies department at the Fraunhofer Institute for Integrated Circuits (IIS) — home of the MP3 and H.264, among other crucial digital content creation technologies — you can imagine Siegfried Foessel has been taking a very close look at techniques for capturing and displaying high-frame-rate (HFR) movies. With discussions still very much open about the benefits of 48 fps versus 60 fps, as well as the issues of converting HFR productions for exhibition at lower frame rates, there are still plenty of questions about best practices for exploiting the increased temporal resolution of high-frame-rate capture. As The Hobbit begins screening for critics and audiences worldwide, we asked Foessel to fill us in on what he and his colleagues have learned about HFR acquisition — and why shooting at 120 fps might be the best plan for anyone looking to maximize flexibility for HFR post-production and delivery.
 
StudioDaily: Tell us about your work on high-frame-rate cinema.
 
Siegfried Foessel: We have been working on this for more than 20 months. We started our investigations with equipment from ARRI [two Alexa cameras running at 120fps] and Stereotec [a mirror stereo rig and an on-set stereographer] and a test shoot in a boxing club at 120 fps.
 
Why did you select that shooting speed?
 
We wanted to check the differences in visual perception between 24, 30, and 60 fps. From 120 fps, we could easily downconvert it to lower frame rates. If you shoot at 120 fps, you can create a virtual 24, 30, or 60 fps rate with different exposure times and different shutter angles by combining two, three, or four images. 
 
What did you learn?
 
We saw that, especially for fast movement, it's quite good to have the higher frame rate. You have less blur and fewer motion artifacts, and it gives you a more realistic impression. On the other hand, we did some tests regarding the bit rate and, especially at higher frame rates, we saw that it makes no difference whether you have 250 Mbps or 500 Mbps. The movement is so fast that you cannot follow it directly.
 
Is 120 fps for HFR acquisition something that's likely to emerge as a de facto industry standard? Or an actual SMPTE standard?
 
120 fps is certainly the best choice for shooting. It's not necessary for projection. It's true that you have the negative of requiring five times the data bandwidth. On the other hand, it makes the processing really easy, and you can set up workflows based on standard post-production tools. If you shot with, let's say, 60 fps, then it's quite difficult to derive 24 fps for older equipment. You have to create some motion compensation, which is a large processing step — and it's not so accurate, because you have to calculate motion compensation for each object. That's why 120 fps will become, more or less, an industry standard. It will not be standardized within SMPTE, because it's a question of the frame rate for capture and SMPTE standards are more for presentation frame rates. There is a working group within SMPTE for HFR, and 24, 30, and 60 fps are all on the road map. Of course, I'm still talking about stereo 3D.
 
Is there any reason why HFR wouldn't be beneficial for flat presentations?
 
The most issues today come with stereo 3D. If you have stereo 3D at only 24 fps, you have a huge motion blur and unsharp edges that make 3D difficult to watch. That's why all of the HFR discussion comes from the 3D stereo perspective. Even in 2D, of course, a higher frame rate is good to have. But if you have fast movement in stereo 3D at 24 fps, the question becomes, why do you have 3D at all when you cannot see it? If we first start with HFR 3D, I assume that HFR 2D will come. All of the projection devices that can play back 60 fps per eye in 3D can also play back up to 120 fps in 2D. And then it's only a question of what you want to send to theaters. You might want to have 24 fps for a more cinematic look, or HFR for a more realistic look. 
 
Is there a reason why the road map for high frame rates tops out at 60 fps? Did you find that to be a limit where increased temporal resolution has no benefits for the human visual system?
 
I think some investigations have found that if you go beyond 55 fps, you have reached the maximum of visual perception. We did some tests at NAB where we projected footage with two projectors running at 120 fps per eye, and for me it looks the same as 60 fps per eye. I don't think it's a really big step forward. 
 
What about people's negative reactions to previews of HFR footage? 
 
It will be a decision for the directors of photography. Because you have this higher frame rate, it looks more realistic, it looks more like TV, it looks more sharp, and not all of the DPs will like this. But it will be their decision in the end. It's an artistic decision. 
 
Is there an expectation that the negative reactions will go away as people see more HFR material?
 
Yes, I think it will be temporary. For 100 years, people have been looking at 24 fps movies, which all have this type of judder and cinematic look. I think we will need some time before they understand that this is a different look and accept it.
 
Will tools for frame-rate downconversion from 120 fps be similar to existing technology used in post-production?
 
It's even easier. You have to linearize the data, so you have to de-gamma your content, combine the images, make an average of the image, and re-gamma the footage. Those are standard post-processing steps that are used today in all kinds of movie manipulation. It's not very difficult.
 
So when, let's say, James Cameron steps up to start shooting his Avatar sequels, do you think that will happen at 120 fps?
 
I think this will happen. 120 fps is not a standard, but it's an option that all of the digital cinema camera manufacturers offer. And if you are able to shoot at 120 fps, it's only a question of whether you can handle the greater amount of data. But it gives you much more flexibility in post-production, so yes, I see this coming.