Frame Interpolation

A video consists of many individual frames shown in quick succession. A cinema film usually shows 24 of these per second, while a smartphone video often shows 30 or 60. The more frames per second, the smoother a movement appears to the eye. Frame interpolation is a technique that inserts additional frames into this sequence afterward. These intermediate frames were never filmed but were calculated by a computer program. The program looks at two consecutive frames and estimates what the frame in between would have looked like.

Why smooth motion saves so much computing time

Capturing or calculating frames takes effort. In a computer game, the graphics card has to redraw every single frame. If it only manages 40 frames per second, the game noticeably stutters. An interpolated intermediate frame is considerably cheaper than a completely newly rendered one. That’s why graphics card manufacturers build this technology directly into their drivers.

This is also useful in film restoration. Old recordings sometimes run at 16 or 18 frames per second and therefore appear hectic and jerky. With calculated intermediate frames, they can be brought up to modern frame rates. The same applies to slow motion: if you slow down a normal video significantly, you suddenly have too few frames. Interpolation fills the gaps instead of simply showing each frame twice.

However, there is also criticism. Many viewers find interpolated cinema films to look unnaturally smooth. This effect has its own name: the soap opera effect, because the picture is reminiscent of cheaply produced TV series. Directors regularly complain that TVs enable this feature by default.

How the program guesses the motion

The core of the technique is called motion estimation. The program compares two neighboring frames and searches for where individual image regions have shifted to. A ball in the bottom left in the first frame and in the top right in the second has obviously moved. From this shift, it can be calculated where the ball must have been in the middle. It is then painted into the new intermediate frame at that position.

Older methods calculate this purely mathematically by searching for similar image sections. Modern systems use neural networks for this purpose, i.e. learning programs that have previously seen millions of real videos. They have learned how people, cars, or water typically move. As a result, they make better guesses than pure computational rules.

Nevertheless, it remains guesswork. It becomes difficult when something is hidden in the first frame and becomes visible in the second. The program has no template for this area and must invent it. Typical errors are smeared hands, wavering edges, or briefly flashing artifacts, i.e. visible image defects. Fast cuts and very abrupt movements are especially prone to this.

From the games console to AI video generation

You most often encounter this technology on your TV. Manufacturers market it under names like Motionflow, TruMotion, or Motion Rate. In the settings, it can usually be switched off, often hidden under terms like intermediate frame calculation. Smartphones also use it for slow-motion recordings with very high frame rates.

In the gaming sector, this has been a selling point for several years now. Nvidia calls its technique Frame Generation, AMD refers to it as Fluid Motion Frames. Both insert a calculated frame between two rendered game frames, thereby doubling the displayed frame rate. One drawback: the controls don’t feel any faster as a result, because the intermediate frames contain no new player input. This delay is called latency.

In news reports today, frame interpolation often comes up in connection with AI video generators. Systems like Sora or Runway generate videos from text descriptions. They frequently calculate only a few key frames and fill in the rest via interpolation. This saves considerable computing costs. It’s important to make the distinction: interpolation doesn’t invent new scenes, it only closes gaps between existing frames.