Latent Space

Computers can only calculate with numbers. A program meant to process images or sentences must therefore first rewrite them as lists of numbers. Such a list might, for example, consist of 512 numbers that together describe a photo. All the possible lists of this kind together form a space, similar to how all the points on a map form a space. This is exactly what is called the latent space: a kind of map on which every piece of content is a point. “Latent” means “hidden,” because these numbers are never visible from the outside.

Why proximity on this map has meaning

The decisive factor is not the conversion itself, but its arrangement. A well-trained program places photos of cats all in the same region of the map. Dogs lie nearby, trucks far away. This suddenly makes a difficult question simple: instead of comparing images pixel by pixel, you only measure the distance between two points.

Many functions used daily rely on this. An image search finds similar motifs by looking for neighboring points. A music app recommends songs whose points lie close to what you just listened to. Translations also benefit from this: “Hund” and “dog” can occupy almost the same point in the same space.

A second advantage is size. A photo with a million pixels can be represented as a list of just a few hundred numbers. The program then has to compute significantly less. In image generators, this is precisely why they can run on ordinary graphics cards at all.

How a model creates this map

No one sets the coordinates by hand. They emerge during training, that is, in the learning phase in which the model works through millions of examples. A typical setup consists of two parts: an encoder compresses the image into the short list of numbers, a decoder builds an image back out of it. At the start, the result is garbage, and the model is corrected for every mistake.

Because the list is so short, the model cannot retain everything. It has to decide which information is important. This very constraint creates the order: features like “fur,” “four legs,” or “night” are worth storing, but the exact color of every single pixel is not. What ends up located where is a byproduct of training and is barely readable to humans.

A common misconception is that every number in the list has a clear meaning, such as “age” or “brightness.” Usually the meaning is spread across many numbers. Sometimes, however, directions with a recognizable effect can be found: moving a facial point in a certain direction makes the face look older. Such points can also be computed with one another, leading to effects like “king minus man plus woman equals queen.”

Latent spaces in image generators and search engines

The term is best known through image generators like Stable Diffusion. The word “latent” is even part of the technical name of the method. These programs don’t paint pixels directly; instead, they first generate a point in the latent space. Only afterward does a decoder translate this point into a visible image. When a video slowly morphs from one motif into another, often only a single point is moving across the map in the background.

In companies, the latent space appears under the keyword vector database. This is a storage system that collects such lists of numbers and quickly finds the nearest ones. Companies store their manuals and support emails there. A chatbot then searches for matching text passages and answers using them instead of from memory.

In news reports, one usually encounters the English form “Latent Space.” Related but not identical is the term embedding: that is a single point in this space, whereas the latent space is the totality of all possible points.