Gaussian Splatting

Gaussian Splatting is a technique for building a spatial representation of a place from ordinary photos. You photograph an object or a room from many angles. A program then computes from this a scene that can afterward be freely orbited on screen, even from perspectives that were never photographed. This scene does not consist of a fixed surface, but of millions of tiny, soft color blobs floating in space. Each of these blobs has a position, a size, a shape, and a color. Together, from a distance, they form an image that is barely distinguishable from a photograph.

Why photorealism suddenly runs in real time

Until now, there were two paths to a 3D scene, and both had a catch. The classic path is models made of triangles, as in computer games. They run fast, but usually have to be built by hand and often look artificial for real places. The second path was so-called neural radiance fields, or NeRF for short: learning programs that reconstruct a scene. Their results looked very realistic, but computing a single image often took seconds to minutes.

Gaussian Splatting resolves exactly this conflict. The quality is on par with the best NeRF techniques. But the rendering runs smoothly, often at over 60 frames per second on an ordinary graphics card. This turns a research demo into something you can use in an app or a VR headset. The technique was introduced in 2023 and has since spread very quickly in research and industry.

What is economically interesting above all is how cheap the capture process is. A few photos or a video from a smartphone are enough for a usable splat. You don’t need a laser scanner or a photo studio. Real estate companies, museums, and car manufacturers can use it to digitize real objects without paying a team of 3D graphic artists.

From photos to millions of color blobs

First, the computer needs to know where the camera was positioned for each photo. It calculates this by comparing recognizable points across several images, such as a door handle or a corner. From these correspondences emerge the camera positions and a rough point cloud. This point cloud is the starting point: each point becomes a small color blob.

These blobs are called Gaussian distributions, because their brightness fades softly from the center outward, like a blurred dab of paint. That is exactly what the word splat means, i.e., a blob. Each blob can be stretched and rotated, so that it can also reproduce a flat wall or a thin branch. The program now renders the scene experimentally from a known camera position and compares the result with the actual photo.

From the difference between the computed image and the photo, it follows how the blobs need to be shifted, recolored, or reshaped. This comparison repeats tens of thousands of times. Along the way, the technique deletes superfluous blobs and splits ones that are too coarse into several smaller ones. After a few minutes to hours, the finished scene is ready. Importantly: in the end, there is no real surface contained within it, only a very good imitation made of color.

Splats in maps, films, and headsets

Gaussian splats are most visible in map services and virtual tours. Instead of jumping between individual panoramic images, you glide seamlessly through an apartment or a street. Museums also use the technique to make exhibits explorable online. In VR headsets, such scenes feel spatially more convincing than any video.

In the film and advertising industry, splats serve as digital backdrops. A team scans a real filming location once and can later plan camera moves on a computer. Robotics companies work similarly: they train control software in photorealistic replicas of real warehouses.

In news reports, the term often appears alongside NeRF, and occasionally as its competitor. A common misconception is that Gaussian Splatting is artificial intelligence in the sense of a language model. It does learn from data, but it does not invent content. A second misconception concerns reuse: anyone who needs an editable model for a 3D printer must first laboriously convert the splat into a surface made of triangles.