Multi-Token Prediction

Programs like ChatGPT generate text piece by piece. To do this, they break language down into small units called tokens: sometimes a whole word, sometimes a syllable, sometimes a punctuation mark. Usually, the program only ever predicts the next of these pieces, appends it to the text, and starts over. Multi-token prediction breaks with this rule. Here, the program learns to guess several upcoming pieces at the same time, for example the next four instead of just one. What used to be one step per piece of text can thus become several pieces of text per step.

Why looking further ahead helps

The obvious benefit is speed. Every single prediction step costs computing time on expensive graphics chips. If a model delivers two or three usable pieces of text per step, the number of steps drops accordingly. Depending on the task, providers report speedups by a factor of 1.5 to 3. For a service with millions of requests per day, that translates directly into noticeable savings.

The second benefit is less visible but more interesting to experts. A model that only has to predict the next piece can often get away with relying on local patterns. After “good” often comes “morning.” But a model that has to hit four pieces in advance needs a rough idea of how the sentence will continue. This forced look-ahead acts during training like an additional training signal.

The effect is especially pronounced in programming and multi-step reasoning. There, later characters depend heavily on a structure that must be planned in advance. Research findings from Meta and the model reports from DeepSeek cite exactly these areas as the biggest beneficiaries.

Several heads on one body

Technically, the model gets additional output layers, often called “heads.” The main part of the model processes the text so far as usual. Head one then predicts the immediately next token, head two the one after that, head three the token after that. All heads share the same expensive groundwork and therefore cost hardly any extra computing time. During training, all predictions are evaluated together and the model is corrected accordingly.

In later deployment, there are two variants. Some systems use the additional heads only during training and discard them afterward. The model then remains normal, but has gotten better. Others keep the heads and use them for advance guessing. The model proposes several tokens at once, and a verification step checks whether the same tokens would also have resulted from normal computation.

This verification is the crucial point. If the suggestions match, they are all accepted. If the third suggestion deviates, everything from that point on is discarded and recalculated. The result therefore remains identical to the slow method, only the path there is shorter. A related technique is called speculative decoding; there, a small auxiliary model supplies the suggestions instead of dedicated heads.

Where the technology stands today

Multi-token prediction became well known in 2024 through a research paper by Meta and through DeepSeek’s models, which documented the method openly. Since then, the term has appeared regularly in technical reports on new models, usually in the section on training objectives or inference speed. When a provider advertises particularly high output speeds, some form of advance computation is often behind it.

In everyday use, you only notice this indirectly. A chatbot whose answer appears in small bursts rather than steadily word by word may be working with pre-guessed tokens. Code assistants in programming environments also benefit, because they can suggest entire lines instead of individual characters.

A common misconception is that the method makes models smarter in the sense of having more knowledge. It does not. It changes how the model is trained and how fast it computes. And it has limits: the further ahead a head looks into the future, the less certain its prediction becomes. That’s why most systems work with only two to four steps of look-ahead.