Autoregressive Model

An autoregressive model is a computer program that builds up its output in small steps. It only ever predicts the next element, for example the next word of a sentence. It appends this word to what has been written so far. Then it starts the same calculation again, now with the extended text as input. In this way, a whole answer emerges step by step. The name comes from the fact that the program refers back to its own earlier outputs: “auto” for self, “regressive” for going back.

Why chatbots answer word by word

This principle explains a characteristic familiar to anyone who has worked with ChatGPT or similar services. The answer doesn’t appear all at once, but trickles onto the screen word by word. This is not just a visual effect. At that moment, the text does in fact not yet exist in full—it is currently being generated.

The procedure also explains how costs arise. Each individual word means a complete computational pass through the model. An answer with 500 words therefore costs roughly 500 times as much computing time as a single word. That is why providers bill their prices according to the amount of text rather than per request.

A second point is more important than it sounds. The model does not plan its answer in advance. It decides anew at each step what fits best. The fact that a sensible structure often still emerges in the end is because a good beginning makes the appropriate continuations likely.

Probabilities instead of fixed answers

The model first breaks text down into small units. These units are called tokens and often correspond to a short word or a syllable. For each position, the model then calculates a probability for every token in its vocabulary. Given “The capital of France is,” “Paris” gets a very high value, “butter” a very low one.

One of these tokens is then selected. The most probable one is not always chosen, since that leads to stiff, repetitive texts. Instead, a kind of dice roll is used, in which more probable tokens are drawn more often. A setting called temperature controls how strongly randomness plays a role. That is why the same model does not give exactly the same answer to the same question every time.

The model learned these probabilities from enormous amounts of text. The next word was hidden each time, and the model was made to guess it. One drawback of this method: an error in the third sentence remains and influences everything that follows. The model cannot go back and correct the beginning.

Not just text: music, code, stock prices

All well-known large language models work autoregressively, such as OpenAI’s GPT series, Google's Gemini, or Meta's Llama. The abbreviation GPT stands for “Generative Pretrained Transformer,” where “generative” refers precisely to this generating of something new. Programming aids such as GitHub Copilot also work this way, writing on line by line of code.

The term is older than the AI wave. In statistics, autoregressive models have been used for decades for time series. One example is a price forecast that estimates the next day from the values of the last few days. The principle is the same, only the type of data differs.

For the sake of contrast, it’s worth looking at image generators such as Midjourney or Stable Diffusion. Most of them are not autoregressive but diffusion models. They start with a noise image and sharpen the entire image over several rounds simultaneously. Research is currently discussing whether a similar approach could also be faster for text than the word-by-word method.