Diffusion Language Models

Most language models powering chatbots today write text like a human at a typewriter: one word after another, left to right, with every decision final. Diffusion Language Models do it differently. They start with a pile of random noise — meaningless data — and refine that pile over many small steps into meaningful text. This procedure is called diffusion and first became known for images, for instance in image generators like Stable Diffusion. Now research is transferring this principle to language. The result are models that develop a text not sequentially, but in a sense out of the whole at once.

Why the word-by-word principle is a drawback

Classic language models cannot backtrack. If the model chooses a phrasing in sentence 1 that fits poorly with sentence 5, that choice is already locked in. The model can no longer go back and adjust the beginning. This approach is called “autoregressive generation” — every new token (i.e., the smallest unit, such as a word or word fragment) depends on all previous ones.

Diffusion Language Models are free of this constraint. Because they refine the text as a whole iteratively — that is, over multiple passes — early decisions can still be corrected in later steps. This gives the model more freedom when planning longer, coherent texts. For tasks where the end of a text needs to influence the beginning — such as legal writing or code — this is a real advantage.

From noise to text — the diffusion process

During training, a Diffusion Language Model learns two things at once. In one step, real text is taken and progressively degraded by replacing or corrupting words with random values — this is called the “forward process.” In the other step, the neural network (a web of mathematical computation units) learns to reverse this corruption. In other words, it learns: what does this damaged text look like, and which word was probably originally there?

When generating new text, the process is reversed. The model starts from a completely random state and applies the learned correction step many times in succession. After enough steps — typically between 10 and 1000 — the noise has turned into coherent, readable text. How quickly and how many steps are needed is one of the central research questions: too few steps yield poor text, too many make the model slow.

Language poses a particular challenge here that does not exist for images. Pixel values in an image are continuous numbers — one can easily compute an intermediate value between red and blue. Words, by contrast, are discrete: there is no word in between “dog” and “cat.” This property makes diffusion technically harder for language than for images, and various research groups are currently developing different approaches to address it.

Diffusion Language Models in current products and research

In everyday life, Diffusion Language Models are barely noticeable yet — commercial chatbots like ChatGPT or Gemini still use the autoregressive approach. But that could change: the model “Mercury,” released in 2025 by the startup Inception Labs, was one of the first diffusion language models with a commercial focus. It is marketed as being able to generate text significantly faster than comparable classic models, because it can compute multiple words simultaneously instead of strictly one after another.

The topic is also active in academic research. Groups at universities such as Stanford and Carnegie Mellon, as well as companies like Meta and Google, have presented their own architectures in recent years. One well-known research model is called “MDLM” (Masked Diffusion Language Model). It uses a variant in which words are simply masked — i.e., made invisible — in the forward process instead of being replaced with numerical noise. This makes the process more linguistically tangible.

Anyone reading tech news often encounters the term in connection with the question of whether diffusion will be the next big wave after GPT-like models. Some researchers believe that processing entire passages of text simultaneously will lead to better and cheaper models in the long run. Others argue that the lead autoregressive models have built up after years of intensive optimization is hardly catchable. The debate is ongoing — the term will play a role in trade articles and product announcements in the coming years.