Deep Learning Super Sampling

A video game has to render every single frame anew, often 60 times per second. The more pixels a screen has, the more work this is for the graphics card. Deep Learning Super Sampling turns this equation around: the game is deliberately rendered at a lower resolution, meaning with fewer pixels. Afterward, a program upscales the image back to the full size of the screen. This program has previously learned from a great many sample images what a sharp image needs to look like. The result appears almost as good as a real, fully rendered image — but costs far less computing time.

What players gain from this

The most important gain is the frame rate, meaning the number of images per second. A game that only manages 35 frames per second at full resolution can reach 70 or more with this technique. Below about 30 frames per second, a game feels choppy and sluggish to respond to. The jump above that threshold is therefore not a luxury, but decides whether a game is comfortably playable at all.

This becomes especially important with very demanding graphics effects. Ray tracing, for example, calculates light, shadows, and reflections with physical precision, and as a result consumes enormous amounts of processing power. Without an upscaling technique, such effects would hardly be usable on ordinary graphics cards. Many modern games now quietly assume that players will enable this technology.

There is also an economic aspect. Anyone who owns an older graphics card can often postpone an expensive new purchase by a year or two thanks to this technique. For Nvidia, the technology is at the same time a selling point, since it only runs on its own cards. Competitors such as AMD and Intel have therefore developed their own techniques.

From a small image to a sharp image

At its core lies a neural network, that is, a program that learns patterns from examples instead of following fixed rules. Nvidia trained it using pairs of images: once the coarse version of a game scene, and once the extremely high-quality rendered version of the same scene. From millions of such pairs, the network learns how to fill in missing details. Edges are smoothed out, and textures are rendered sharp again.

Crucially, the network does not just see the current frame. It also receives the previous frames and so-called motion vectors. These indicate how far each object has moved since the last frame. This allows the technique to reuse details from earlier frames and place them correctly.

For this computation, Nvidia cards use special processing units called Tensor cores. They are tailored to the type of computation neural networks require and work in parallel to normal graphics rendering. Newer versions additionally generate entirely new intermediate frames that never existed in the game at all. This further increases the frame rate, but can slightly increase input lag — a well-known point of criticism.

In the graphics menu and in the quarterly figures

The term is most often encountered in the settings menu of games. There, one finds levels such as Quality, Balanced, or Performance. These determine how small the originally rendered image is allowed to be. The higher the performance level, the smoother the game runs, and the more likely one is to notice flaws such as flickering edges or smeared details during fast motion.

The technology also constantly appears in reviews and buying guides. A common misconception is that the technique magically makes images better than the original. It reconstructs information that is actually missing — and sometimes gets it wrong. In still images the result often looks excellent, while in fast-moving scenes weaknesses tend to stand out more.

Finally, the term also plays a role in business news. It is considered an example of how AI computing power is migrating from the data center into end devices. Similar upscaling techniques are now found in game consoles, televisions, and smartphone cameras.