Rotoscoping

Rotoscoping refers to isolating characters or objects in a video — frame by frame, meaning for every single frame. The result is a precise mask that shows exactly which pixels belong to the character and which belong to the background. This mask can then be used to replace the background, insert special effects, or place a character into a completely different scene. In the past, artists traced outlines by hand onto celluloid — frame by frame, for hours on end. Today, AI systems automatically handle most of this work.

Rotoscoping as the foundation of visual effects

Without clean cutouts, many effects in film and television would be impossible. When an actor is filmed in front of a green screen, the camera’s background can later be replaced digitally. But not every scene can be shot under such controlled conditions. Sometimes a character has to be isolated from real street footage — in the middle of a busy park, under changing light, with hair blowing in the wind.

This is exactly where rotoscoping becomes indispensable. It provides the precise masks on which compositing — the process of combining different image layers — is built. A poorly isolated arm or a pixelated hairline edge immediately catches the viewer’s eye. The quality of rotoscoping therefore directly determines how believable an effect appears.

How AI takes over the work of cutout artists

Traditionally, a person marks the outline of a character in a single frame using a mouse. For a close-up shot, this can quickly add up to hundreds of points — and a film has 24 frames per second. For a ten-minute clip featuring a moving person, this adds up to over 14,000 individual frames.

Modern AI tools, for example from Adobe or the research lab Meta, analyze such a clip using neural networks — that is, systems trained on millions of sample images to recognize objects. The user clicks once on the person, and the model automatically tracks them through all frames. It detects motion, depth, and body boundaries and adjusts the mask on its own. What used to take days now takes minutes.

Difficult cases, however, still remain. Very fine hair, transparent fabrics, or two people overlapping each other still frequently overwhelm current models. In professional productions, teams therefore combine AI-generated drafts with manual corrections.

Rotoscoping in products and current debates

Anyone who blurs or swaps out the background of a video on a smartphone is using automatic real-time rotoscoping. Apple, Google, and Samsung have built corresponding features directly into their camera apps. Video conferencing software like Zoom or Teams works the same way: the person is separated from the apartment behind them, without needing a green screen.

In the film industry, rotoscoping is a central topic in the current debate about AI and workers' rights. The 2023 Hollywood strike was partly about whether studios should be allowed to use AI tools to replace rotoscoping artists. The union SAG-AFTRA ultimately negotiated rules that tie certain uses of AI to consent and compensation.

Rotoscoping is thus a prime example of how an AI technology can simultaneously enable useful everyday functions while raising profound distributional questions in professional industries.