Perplexity

Programs like ChatGPT work by constantly guessing which word comes next. In doing so, they don’t just make a single guess but distribute probabilities across many possible words. Perplexity measures how well these guesses match a real text. The program is given a text it has never seen before, and word by word one checks how confidently it expected the word that actually follows. If it is usually confident and correct, perplexity is low. If it is often surprised, it is high. You can roughly picture the value like this: a perplexity of 10 means the program was, on average, as uncertain as if it had to choose between ten equally good options for every word.

Why researchers watch for low values

When building a language model, you need a number that tells you whether a change helped. This is exactly what perplexity is used for. It can be calculated automatically, without a human having to judge the answers. That’s why it appears in almost every research paper on language models, often in tables covering several test texts.

The number is also very sensitive. Even small improvements to a model’s architecture or training data show up immediately. A team can thus compare hundreds of variants without having to organize an elaborate human review each time. Historically, the decline in perplexity was the key evidence that larger models with more data actually get better.

One important distinction: perplexity measures language prediction, not usefulness. A model with very good perplexity can still answer rudely, invent sources, or ignore instructions. Other tests exist for such properties, for instance task collections with right and wrong answers, or evaluations by human testers.

How the number is derived

The model is given a test text and predicts, for every position, which word follows. For the word that is actually there, the predicted probability is recorded. For a thousand words, there are a thousand such numbers. Perplexity is calculated from the average of these values. Roughly speaking, it is the reciprocal of the average hit probability.

An example makes this tangible. In the sentence “I’m taking the bus to ...”, “school” is quite obvious. If the model assigns this word 50 percent, it is hardly at a loss at this point. If, on the other hand, a text contains a name nobody could guess, the probability drops to a tiny value. Such spots drive perplexity upward.

A common misconception is that perplexity values from different publications can be compared directly. Usually this doesn’t work. The number depends on the test text and on how the model breaks text into building blocks. Some systems work with whole words, others with word pieces or individual characters. Values are only comparable if the test text and the tokenization are identical.

Perplexity in papers and products

In everyday life, nobody encounters this number directly. No chatbot displays its perplexity. Yet it lies behind many announcements about new models. When a company writes that its compressed model is “practically equivalent,” this is often based on a perplexity value that barely rose.

This is especially important when shrinking models. In that process, the internal numbers are stored more coarsely so that less memory and computing time are needed. Developers then check perplexity to see how much quality was lost. An increase from 6.2 to 6.4 is considered harmless, a jump to 12 a catastrophe.

Incidentally, the word also shows up as a company name: the search engine Perplexity AI named itself after the metric. So anyone coming across the term in the news should briefly check whether the metric or the company is meant. In technical texts, the abbreviation PPL is usually used, often together with the name of the test text.