Autoregressive Prior
An autoregressive prior is an auxiliary model that predicts the compressed internal description of data piece by piece: each value is estimated from the values already known. Such models are found in image generators, in speech synthesis, and in modern compression methods.
Many AI systems do not store an image or a sound as raw data, but as a short sequence of numbers, a kind of short description. To generate new content from this, the system must know which number sequences are actually meaningful and which merely produce noise. This exact expectation is called a prior in statistics, that is, a prior assumption about what is likely. An autoregressive prior builds up this assumption step by step: it predicts each value of the sequence from the previous values. The word autoregressive means exactly that — the model draws on its own previous output. You can picture it like a sentence in which each word is inferred from the beginning of the sentence.
Why short descriptions need an expectation
A model that compresses images, for example, produces a grid of 32 by 32 codes. Purely mathematically, there are astronomically many combinations for this. But only a tiny fraction of them result in an image that looks like a photo. Without a learned expectation, a system would therefore almost always produce nonsense.
The autoregressive prior closes exactly this gap. It learns from many examples which codes typically stand next to each other. A blue patch of sky is more likely to follow another blue patch of sky than a piece of car tire. When generating new content, sampling is therefore drawn only from the plausible region.
A second reason is compression. Whoever can predict well what comes next needs few bits to store the deviation. Modern image and video compression uses this principle directly. The better the prior, the smaller the file at the same quality.
Prediction step by step
The procedure usually runs in two stages. First, an encoder learns to translate data into a short code sequence, and a decoder learns to build an image back from it. Only afterward is a second model trained on these code sequences. This second model is the prior.
During training, the prior is given a portion of the sequence and is supposed to guess the next value. It does not output a fixed answer, but a probability distribution over all possible values. The error between prediction and actual value is measured and the weights are adjusted. After many passes, the model knows the typical patterns.
During generation, the process is reversed. The model draws a value from its distribution, appends it to the sequence, and predicts the next one. This is how the complete code sequence is created, which the decoder then turns into an image. The downside is obvious: because each step builds on the previous one, generation can hardly be parallelized. This is exactly where diffusion models have an advantage, refining an entire output simultaneously instead of building it up value by value.
In image generators, speech synthesis, and codecs
The principle became well known through models such as VQ-VAE-2 and the first version of DALL·E. Both generated images by having an autoregressive model write a code sequence, which was then translated into pixels. The technique is also widespread in speech synthesis: systems predict sound segments one after another, thereby producing natural-sounding voices.
In the news, the term often comes up in connection with neural compression. Companies advertise storing video at the same image quality significantly smaller than classical methods. However, the computational cost for this is high, which is why such codecs have so far rarely run on ordinary devices.
A common misconception is to confuse the autoregressive prior with the actual generator. The prior does not produce images, only the code sequence. Only the decoder turns this into something visible. Both parts are trained separately and can also be swapped out separately.