Diffusion Model

A diffusion model is a program that generates new images. It doesn’t start with a blank canvas, but with pure image noise: a picture made up entirely of random colored dots, like the static on an old television. From this chaos, the program removes a bit of randomness in many small steps. After a few dozen passes, a clear motif remains that never existed before. Which motif emerges is controlled by a short description in plain language. The technique also works for audio and short videos, but it became famous through images.

Why image generators suddenly started working

Before 2021, computer-generated images were usually immediately recognizable as such. Distorted faces, blurry backgrounds, and strange color gradients were the norm. With diffusion models, this changed within a few months. Programs like Stable Diffusion, Midjourney, and DALL·E are all based on this principle. They turned a research topic into a tool used by millions of people in their browsers.

The economic impact is significant because images cost money in many industries. Advertising agencies, game companies, and publishers constantly need new visual material. A draft that used to require a photo shoot or an illustration commission is now created in seconds. This drives down prices and changes job profiles. At the same time, new tasks are emerging, such as formulating good image descriptions and post-processing the results.

The technology is controversial for two main reasons. First, the models were trained on huge amounts of images from the internet, often without the consent of the creators. Several lawsuits by artists and image agencies are pending as a result. Second, it can be used to create deceptively realistic photos of events that never happened. Such fabrications are called deepfakes, and they are a problem for election campaigns and news coverage.

From noise back to image

Training runs in reverse to the later application. Real photos are taken and random noise is poured over them step by step until only noise remains. For each of these stages, the system therefore knows both states: the slightly noisy image and the somewhat cleaner one before it. From this, it learns exactly one ability: to estimate, from a noisy image, which part of it is disturbance. This ability is later applied millions of times.

In image generation, the process is reversed. The model receives random noise and removes part of it. The result is fed back in, more is removed again, and so on. Typically, twenty to fifty such steps are involved. You can imagine it like a sculptor removing material from a block of marble until a figure becomes visible.

The text description influences every single step. It is translated into numbers and guides the decision as to what counts as noise and what belongs to the motif. A common misconception is that the model searches a database for matching images and assembles them. It does not do that: it does not store any images, only the learned rule for denoising. That’s why motifs are created that it has never seen during training.

Where diffusion is used

The technology is most directly encountered in image generators. Adobe Photoshop uses it to fill in missing parts of images or remove objects. In smartphone cameras, related methods improve photos in low light. The image functions in chat programs like ChatGPT or Gemini also work on this principle at their core.

In business news, the term usually comes up in connection with Stability AI, the company behind Stable Diffusion, or with copyright lawsuits. A second recurring topic is power consumption: each of the many computation steps costs computing time on expensive graphics cards. That’s why companies are working on getting by with fewer steps.

It’s worth distinguishing this from language models like GPT. These generate text word by word from left to right. A diffusion model works on the entire image at once and refines it round by round. Both are AI models, but their working principle is fundamentally different. Recently, diffusion has also been used for videos, for example in OpenAI’s Sora, where the sequence of images additionally has to match up over time.