3D Gaussian Splatting

3D Gaussian Splatting is a technique for creating a walkable 3D scene from ordinary photos. You photograph a room or an object from many angles, often with a phone. A program then computes a representation from this that you can freely orbit around on screen. This representation does not consist of solid surfaces, but of millions of tiny, translucent colored blobs in space. Each blob has a position, a size, a color, and a transparency. From a distance, these blobs merge into an image that is barely distinguishable from a photograph.

Why photorealism is suddenly running in real time

Photorealistic 3D scenes already existed before this. The best-known previous technique was called NeRF, short for Neural Radiance Field. In this approach, a neural network learns to predict a color for every point in space. It looks great, but it is slow. A single image requires the network to be queried millions of times.

Gaussian Splatting delivers similar image quality but runs considerably faster. On an ordinary gaming graphics card, a smooth 60 to 100 frames per second are possible. It is precisely this speed that makes the technique interesting for games, for virtual reality, and for browser-based applications. Choppy 3D views quickly become unpleasant, and in VR even physically uncomfortable.

There is also a practical point. Classic 3D models for films or games are built by hand, which takes days or weeks. With splatting, a walkthrough with a camera and some computing time is enough. This lowers costs so dramatically that 3D captures become accessible to small companies and individuals.

From photos to millions of tiny color clouds

The starting point is several dozen to several hundred photos. First, software determines the position and direction from which each photo was taken. This also produces a rough point cloud of the scene. Each of these points then becomes the starting point for a colored blob. The technical term for this is a Gaussian function: a bell-shaped distribution that is strong at the center and fades gently toward the edges.

After that, an optimization process begins. The program renders the scene from the viewpoint of one of the original photos and compares the result with the actual photo. Wherever it deviates, it shifts, colors, stretches, or rotates the blobs slightly. This comparison runs tens of thousands of times. In addition, the system is allowed to split blobs when an area is too blurry, and delete them when they are almost invisible.

A common misconception: splatting does not contain a trained neural network in the final result. What gets optimized are the numerical values of the individual blobs directly. The result is therefore more of a very large file of point data than an AI model. For rendering, the blobs are sorted by distance and layered from back to front.

Real estate, museums, and map apps

So far, the technology is most visible in remote viewings. Real estate agencies present apartments as freely rotatable 3D scenes instead of photo galleries. Museums digitize exhibits so they can be viewed online. Google, too, uses related techniques to generate realistic views of cities and landmarks in Maps.

In industry, the technique is used for documentation. A construction site or a factory hall is filmed and can later be inspected from any angle. Film studios use splatting for backgrounds, because a real location can be scanned faster than it can be rebuilt. In robotics, research is underway into whether such scenes can help machines understand their environment.

There are limits, however. Mirrors, glass, and water often look wrong, because their appearance changes drastically with viewing angle. Moving objects leave streaks. And a single scene can easily be several hundred megabytes in size. In the news, the term therefore usually comes up when a company significantly shrinks these files or renders them smoothly on a phone for the first time.