Latent Space

A computer can’t do anything directly with a photo or a sentence. It only computes with numbers. That’s why an AI system first translates every input into a long list of numbers. This list doesn’t describe the content pixel by pixel, but in condensed form: more like “cat, side view, outdoors, bright” than “point 1 is dark gray.” All possible such lists of numbers together form a space, which is called latent space. Latent means hidden, because this representation isn’t directly readable by humans.

Why proximity in number space is so useful

The crucial point is the arrangement. Content that is similar gets similar lists of numbers and thus lies close together. A photo of a house cat lies near a photo of a lynx and far from a photo of a washing machine. This also applies to language: the sentences “What will the weather be like?” and “Will it rain tomorrow?” end up in almost the same place, even though they share hardly any words.

This makes it possible to measure meaning. You simply calculate the distance between two points and know how similar two pieces of content are. This is exactly what semantic search is based on—that is, searching by meaning rather than by exact words. A support system can thus find the right help article even if the user uses completely different terms than the documentation does.

A second benefit is compression. An image with a million pixels can often be usefully described with just a few hundred numbers. Image generators therefore don’t compute on the finished image at all, but in this smaller representation. This saves an enormous amount of computing time and is one reason why such programs run on ordinary graphics cards today.

How a model learns this representation

No one determines which number stands for which property. The space emerges on its own during training. A common method uses two parts: an encoder compresses the input into the short list of numbers, and a decoder tries to reconstruct the original from it. Because the bottleneck in the middle is very narrow, the model has to discard what’s unimportant and keep what’s essential.

You can think of it like a map. A map leaves out trees and house numbers but preserves the positions of cities relative to one another. Anyone who understands the map can read off distances without ever having been there. The latent space is such a map for meaning, only with hundreds of directions instead of two.

Because it is a continuous space, you can move around within it. If you take the point of one face and the point of another and move step by step between them, intermediate faces emerge. Some directions correspond to surprisingly clear properties, such as “older” or “smiling.” A common misconception, however, is that each individual number stands for one feature. Usually a property is spread across many numbers at once.

Latent space in products and headlines

The principle is most visible in image generators. Stable Diffusion even carries the idea in its name: the model first generates a rough representation in latent space and only converts it into an actual image at the very end. Music and video models work according to the same pattern as well.

In everyday life, the term is usually encountered under other names. When embeddings or vector databases are mentioned, it’s precisely these lists of numbers and their storage that are meant. Companies store their documents as points in such a space so that a chatbot can find the relevant passages before it answers. Recommendations on streaming services are likewise based on placing movies and users within a shared space.

An important distinction: latent space refers to the entire space, while an embedding is a single point within it. And the space is only as good as the training data. If it contains biases, certain professions and certain groups of people end up unintentionally close together. Such distortions are a recurring topic in debates about fair AI.