Post-production with AI

When a film has been shot, it is far from finished. What follows is post-production: everything that happens after filming ends. This includes editing, color grading, sound, music, subtitles, and removing distracting details from the image. Traditionally, people do this using specialized computer programs, often over weeks or months. In post-production with AI, programs take over parts of this work, having previously learned from millions of images and audio recordings what a good result looks and sounds like. Humans still make the decisions, while the program offers suggestions and handles the fiddly details.

Why studios rely on it

Post-production is expensive mainly because it costs working time. A single effects artist can spend days retouching a microphone out of every frame of a scene. A feature film has around 130,000 individual frames per hour. It is precisely with such tedious repetition that trained programs excel: they apply the same correction automatically across thousands of frames.

As a result, who can actually afford to do this kind of work is shifting. A two-person YouTube channel today uses tools that ten years ago were only available to large studios. Automatic subtitles, clean audio tracks, and color correction are built into common editing programs and are often included in a subscription.

At the same time, the technology is controversial in the industry. The 2023 Hollywood strikes were partly about whether studios are allowed to digitally reuse actors' faces and voices. Editors, voice actors, and retouchers also fear that some of their tasks will disappear. This is not a technical problem but a contractual one – and therefore a recurring topic in business news.

From raw footage to finished image

Most of these tools are based on pattern recognition. A program is shown a great many example pairs: noisy footage and clean footage, blurry image and sharp image. From this, it learns how to get from the bad version to the good one. It can then apply this step to material it has never seen before.

A second important building block is object recognition. The program independently marks where in an image a person, a face, or a car appears, and tracks that object throughout the whole scene. Only this makes it possible to swap backgrounds or brighten individual people afterward, without anyone having to trace every single frame by hand.

A common misconception is that the AI shapes the film itself. It plausibly fills in missing details, but it has no taste and no sense of rhythm. That is why a human still chooses the shot during editing. And with invented details, things get risky: an upscaled image can show patterns that were never in the original footage. For documentaries and news, this is a real problem.

From the Netflix catalog to the phone app

The effect is most visible with old films. Black-and-white footage is colorized afterward, scratchy copies are denoised and brought up to high resolution. Streaming services use this to bring classics into their catalog. The restored Beatles recordings of recent years were also made using such techniques, which separate voice and piano from a single old audio track.

In series, dubbing and subtitles are a major field. Programs generate subtitles automatically from the audio and translate them into dozens of languages. There are also tools that recreate a voice in a foreign language and adjust the lip movements to match. How convincing this looks depends heavily on the source material.

Even without a film studio, one encounters this technology constantly. Portrait mode on a smartphone computes the blurred background instead of creating it optically. Video conferences filter keyboard clatter out of the audio. And apps like CapCut or Premiere Pro offer features that automatically cut out pauses in speech. This is post-production with AI in miniature, just without the credits.