Upscaling

Every digital image consists of a grid of tiny colored dots, the pixels. A small image has few of them, a large one many. Upscaling means: turning an image with few dots into an image with many dots. The problem is obvious. The additional dots have to come from somewhere, since they aren’t present in the original. An upscaling method must therefore invent them – in such a way that the result looks natural and doesn’t appear smeared or blocky.

Why blurry should become sharp again

Screens have grown considerably larger in recent years. A modern TV has around 8 million pixels, an old TV recording provides perhaps 400,000. When this old material is shown on the large device, the gap has to be filled. Good upscaling determines whether a film from 1995 looks presentable on a new TV or like a coarse pile of blocks.

The second major reason is computing power in video games. Rendering a game at full resolution costs the graphics card a great deal of work. If it instead renders the image at half width and height and then scales it up, it only needs to generate about a quarter of the pixels. This can double the frame rate. That’s exactly why upscaling is built into almost every major game title today.

There are also applications where beauty isn’t the point. Satellite images, medical scans, or surveillance videos are upscaled to make structures more recognizable. However, a serious risk lurks here, which will be addressed further below.

From doubling pixels to learned guessing

The simplest method copies each dot multiple times. The image becomes larger, but also jagged and coarse. Somewhat better are methods that form averages between neighboring dots. The jagged edges disappear, but everything becomes slightly blurry. Both methods generate no new information, they merely redistribute the old.

AI upscalers work differently. A neural network – a program that learns patterns from examples – is shown millions of image pairs. Each pair consists of a sharp photo and the same shot artificially shrunk. The network learns in the process how a blurry edge typically looks sharp, how skin, hair, or brick walls are actually structured. Later, it receives an unknown small image and applies this knowledge.

With videos, there’s an additional trick. Consecutive frames show almost the same subject, just shifted minimally. From several such frames, more detail can be assembled than from a single one alone. Game upscalers like Nvidia's DLSS or AMD's FSR additionally use data from the game engine, such as which direction objects are moving. What matters at the core is this: the result is a well-founded guess, not a reconstruction of the truth.

Between the TV menu, the graphics card, and the courtroom

In everyday life, you usually encounter upscaling without noticing. Almost every TV has it built in and advertises it with terms like “AI picture optimization.” Streaming services upscale older shows before offering them as “remastered.” Smartphones use it for digital zoom, so that a zoomed-in subject doesn’t immediately turn mushy. Image editing programs, too, now offer an upscaling button.

In tech news, the term mainly appears around graphics cards. When a manufacturer introduces a new generation, the frame rates cited are often measured with upscaling switched on. Anyone reading reviews should therefore pay attention to whether a figure was achieved with or without this technology. The difference can easily be around 50 percent.

A common misconception, borrowed from crime shows, is the idea that you can “compute” a pixelated license plate into sharpness and thereby read it. That doesn’t work. The upscaler invents a digit that might fit the blur – it could be completely wrong. Such images are therefore not evidence. Upscaling improves the impression, not the information content.