Next-Token Prediction

Programs like ChatGPT appear as if they were forming a thought and then writing it down. In reality, they do something much simpler. They look at the text so far and guess which piece of it comes next. Such pieces of text are called tokens: sometimes a whole word, sometimes just a syllable or a punctuation mark. The guessed piece is appended to the text, then the guessing starts over. This exact step-by-step continuation of writing is called next-token prediction.

Why guessing turns into a skill

At first glance, this sounds like a rather pitiful ability. Someone who only guesses the next word shouldn’t be able to solve a math problem. The point is: to guess really well, one has to know an astonishing amount. If the text reads “The capital of Norway is”, only a single word is correct. Anyone who reliably types “Oslo” here has learned geography without anyone having taught them geography.

The same applies to grammar, forms of politeness, programming languages, and patterns of argumentation. All of this is contained within the one goal of continuing the text as plausibly as possible. That’s why a single training task suffices for a model that later does all sorts of different things. There’s no need for handcrafted examples for every single skill.

This is exactly where the economic appeal lies as well. Training material is simply text, and text exists on the internet in vast quantities. Nobody has to label or sort it beforehand. This combination of a simple task and nearly unlimited material is a major reason for the AI boom of recent years.

From probability distribution to finished sentence

The model never names just a single answer. It calculates a probability for every possible token. After “I am going to the”, “doctor” might come in at twelve percent, “station” at eight, “window” at a fraction of that. One option is then selected from this list.

How bold this selection turns out to be is controlled by a value called temperature. At low temperature, the model almost always picks the most probable suggestion. The answers become reliable, but also somewhat boring and repetitive. At high temperature, it reaches for less probable words more often. This sounds more creative, but tips over into nonsense more quickly.

During training, it works the other way around. Real text is taken, the next token is covered up, and the model is made to guess. If the prediction deviates from the actual word, the internal settings are corrected minimally. This happens billions of times over. A common misconception is that the model memorizes the training texts and later copies them out. What gets stored are not sentences, but patterns.

What this means for chatbot answers

You can see this principle every time a chatbot types out its answer word by word on the screen. This is not a visual effect. The text really is generated piece by piece, and each piece requires a new round of computation. That’s why long answers take longer than short ones and also cost the provider more money.

This principle also gives rise to the best-known weakness of these systems. A model looks for the most plausible next token, not the truest one. If a made-up source citation sounds linguistically convincing, it gets output. Such confidently delivered false statements are called hallucinations. They are not a bug that can simply be programmed away, but a direct consequence of the way these systems are built.

In news coverage about AI, the term usually surfaces within one particular debate: is mere continuation of text enough for genuine understanding? Critics dismissively speak of a “stochastic parrot”. Proponents point out that the models solve tasks that never appeared in the training text in that form. The question remains unresolved.