Diffusion Model

A diffusion model is a computer program that generates an image from a text description. It doesn’t start with a blank canvas, but with pure image noise. Noise is a swarm of random colorful dots, like the static on an old TV without reception. From this static, the program removes more and more randomness in many small steps. After fifty or a hundred passes, a clear subject remains. The name comes from physics: diffusion describes how a drop of ink spreads in water until nothing is recognizable anymore. The model has learned to run exactly this process in reverse.

Why this technique has conquered image generators

Before 2021, artificial images were mostly created using a different method. In it, two networks competed against each other: one painted, the other tried to detect forgeries. This method often delivered good results but was hard to tame. It frequently produced the same motifs over and over or collapsed during training.

Diffusion models are much more well-behaved. They learn stably, cover a wide variety of subjects, and can be controlled well via text. That’s why practically all well-known image generators are built on them: Stable Diffusion, Midjourney, and DALL·E work on this principle. Video generators like Sora and many tools for music and speech synthesis now use it as well.

Economically, this has become a billion-dollar market. Advertising agencies, game makers, and film studios are using it to replace parts of their image production. At the same time, lawsuits are underway because the models were trained on images from the internet, often without the creators' permission. Anyone reading news about AI art is almost always reading about diffusion models.

From noise back to the image

Training begins with real photos. To each photo, the computer gradually adds noise until only grit remains. At each of these steps, the model memorizes exactly what noise was added. The actual task, then, is: look at a noisy image and say which part of it is disturbance.

During generation, the process is reversed. The model receives pure noise and estimates which part of it needs to go. It subtracts this part and obtains a slightly clearer image. It repeats this step many times. A subject gradually emerges from the randomness, much like a sculptor freeing a figure from a block of stone.

For the right thing to emerge in this process, the text prompt comes into play. A second model translates the sentence “a red fox in the snow” into numbers. These numbers guide the decision at every step as to what counts as disturbance and what counts as subject. A common misconception, by the way, is that the model glues together image parts from a database. It doesn’t: it doesn’t store any images, only rules about what images typically look like.

Spotting diffusion models in everyday life

They are most visible in apps that turn a sentence into an image. The “remove distracting object” feature in smartphone photo galleries often works this way too. The model fills the gap with a motif that matches the surroundings. Tools that extend a photo beyond its edge work similarly.

Two points keep coming up in the news. First, power consumption: each image requires many computational steps in sequence, which strains data centers. Second, forgeries, since realistic images of people and events can now be created in seconds. That’s why providers are working on invisible watermarks that reveal that an image originated from a machine.

Anyone who experiments with it themselves quickly notices the typical weaknesses. Text on signs often turns into a jumble of letters, and hands sometimes end up with six fingers. The reason is that the model attends to plausible patterns rather than to countable rules. Newer versions have gotten better at this, but none is perfect.