Kullback-Leibler Divergence

Many predictions are not fixed answers but probabilities. A weather service doesn’t say “it will rain,” but rather “70 percent chance of rain, 30 percent dry.” Such a list of probabilities for all possible cases is called a distribution. The Kullback-Leibler divergence is a number that indicates how much two such distributions differ. It is zero when both are exactly identical, and becomes larger the more the prediction deviates from reality. It is named after the mathematicians Solomon Kullback and Richard Leibler, who introduced it in 1951.

Why AI models are measured against this number

A language model predicts at each step which word comes next. It does not output a single answer, but rather probabilities for thousands of words. During training, this prediction is compared with what actually appeared in the training text. Precisely this comparison requires a measure for “how far off.” The KL divergence provides it.

The advantage over a simple right-or-wrong counter lies in the gradation. A model that gives the correct word 40 percent is penalized more mildly than one giving it 2 percent. Both were wrong with their favorite, but the first was clearly closer. This fine gradation allows training to improve step by step.

A second reason is more practical. One often wants a new model not to stray too far from an old one. During fine-tuning with human feedback, for instance, a chatbot should become more polite, but not unlearn its entire linguistic knowledge. To achieve this, one measures the KL divergence between the old and new model and limits it. The number then acts like a leash that keeps the model close to its starting state.

Surprise as a computational quantity

Behind the formula lies the concept of surprise. An event one assigned 99 percent likelihood to barely surprises. One with 1 percent surprises greatly. Mathematically, this is expressed via the logarithm of the probability: the smaller the predicted probability, the greater the surprise value.

The KL divergence now adds up all these surprises, weighted by the true distribution. So one asks: How much more surprised am I on average if I rely on the wrong distribution instead of the correct one? This additional surprise is measured in bits, the same unit as storage space. A value of one bit roughly means: per observation, one additional bit is needed to transmit the same information.

An important quirk trips many people up. The KL divergence is not a true distance, because it is not symmetric. The value from distribution A to B is usually different from B to A. That is why it is called divergence and not distance. In practice, one must always know which distribution plays the “true” role and which is the prediction.

Where the number appears in research and products

In everyday life, one never encounters the KL divergence directly, but certainly its consequences. Every large language model was trained with a procedure closely related to it: the so-called cross-entropy loss. Minimizing this value mathematically means minimizing precisely the KL divergence between the model’s prediction and the training data. The two quantities differ only by an amount the model cannot influence anyway.

In articles and company reports, the term usually appears in the context of fine-tuning. In methods such as RLHF, i.e., training with human ratings, there is almost always a KL term in the objective function. It is also central to knowledge distillation: a small model learns there to imitate the probability distributions of a large model.

Outside AI, the same measure is used in statistics, image compression, and genetics. Everywhere, it concerns the same question: How costly is it to work with a simplified model of the world? Anyone who comes across “KL” in a technical report can usually translate it as follows: here, it is being measured how far apart two predictions have drifted.