Super-Resolution

Every digital image consists of a grid of tiny colored dots, the pixels. The fewer dots an image has, the coarser and blurrier it looks once displayed at a larger size. Super-resolution is the umbrella term for methods that compute a finer version with significantly more dots from such a coarse image. In doing so, nothing that was actually photographed is restored: the new dots are invented by the program, albeit very plausibly. An image with 500 by 500 dots can thus become one with 2000 by 2000 dots. To the eye and on screen, the result often looks like a genuine high-quality recording.

Why pixelated images are an expensive problem

High resolution costs money in many places. A video in 4K requires many times the data volume of a video in HD. Anyone delivering it over the internet pays for every gigabyte transmitted. If a device receives a small video and upscales it itself, the provider saves bandwidth. That is exactly why super-resolution is now built into televisions, graphics cards, and streaming apps.

The second reason concerns recordings that cannot be repeated. Old family photos, historical film footage, or images from a satellite exist only in the quality in which they were originally created. Improving a camera after the fact is impossible, improving the image is not. Archives, film studios, and research groups therefore make intensive use of the technology.

There is, however, an important caveat. Because the new details are computed rather than measured, they must not be blindly trusted. An upscaled surveillance video is not evidence, since a face or a license plate may simply have been guessed incorrectly. Experts therefore speak of plausible completion, not reconstruction.

From simple smoothing to learned detail

The oldest method is called interpolation. Here, the average value is simply inserted between two known points. This works, but it only makes the image bigger and softer, not sharper. Edges then appear blurred rather than crisp.

Today’s methods work with neural networks. These are programs that are not programmed rule by rule, but rather learn from examples. For training, millions of sharp photos are taken and artificially shrunk. The network receives the small image and is supposed to predict the large one. After each attempt, it is compared with the real original and corrected to some extent. After very many rounds, it knows typical patterns: what hair looks like, how brick walls are textured, how letter edges run.

With videos, there is an additional trick. Consecutive frames show almost the same subject, only minimally shifted. Genuine additional information can be extracted from several such frames. The result is more reliable than with a single photo, because the details are in part actually measured rather than merely guessed.

In the television, in the game, and in space

The technology is most commonly encountered in computer games. Methods such as Nvidia's DLSS or AMD's FSR let the graphics card render the game internally at a lower resolution and then upscale it. The image looks almost identical, but the frame rate increases significantly. For gamers, this means smoother display without new hardware.

Smartphones also use super-resolution, for instance in digital zoom. The lens cannot actually zoom in as far as the display promises. The rest is handled by software, which computes a detailed image from several frames captured in quick succession. In televisions, the same principle runs under names such as AI Upscaling.

In the news, the term often comes up in connection with restored films, such as old documentary footage in 4K. It is also technically relevant in medicine and in satellite observation, where finer maps are meant to be produced from few measurement points. Related, but not the same, is image generation: there, an image is created entirely anew from text, whereas super-resolution always requires an existing image as a template.