Image model

An image model is a computer program that generates images. You give it a short piece of text, for example “a red fox in the snow at sunset.” Within a few seconds, it delivers a matching image that has never existed before. So it doesn’t search the internet for this image, but rather calculates it anew, pixel by pixel. This is possible because the program has previously analyzed hundreds of millions of images together with their captions. In doing so, it learned which image content typically corresponds to which words.

How this is changing image production

Images were expensive for centuries. You needed a photographer, an illustrator, or at least time and practice. An image model pushes these costs down to a few cents per image. Advertising agencies, newspapers, and game companies have long been using this for drafts and backgrounds. For professions that make a living creating simple images, this is a serious economic threat.

At the same time, a trust problem is emerging. A photo was long considered proof that something really happened. This evidentiary power is fading now that anyone can invent a convincing image of an event within seconds. Images of politicians in fabricated situations have already spread millions of times across social networks. That’s why providers are working on invisible watermarks — markings within the image that software can detect but the eye cannot.

On top of that, there is a legal dispute. The training images often come from artists and photographers who were never asked. Several major lawsuits against providers of image models are underway in the US and in Europe. How they turn out will help determine what this technology may look like in the future.

From noise to finished image

Most image models today work on the diffusion principle. During training, a real photo is taken and image noise — random blotches of color — is gradually sprinkled into it. After enough steps, all that remains is a gray pixel mush. The model now practices the opposite: for each intermediate step, it is supposed to estimate which noise was just added. If this estimation is repeated often enough, the process can be reversed.

When generating an image, the model therefore starts with pure random noise. Over roughly twenty to fifty computation steps, it gradually removes the noise from it. The input text steers this process: it guides each step in the direction that matches the description. A motif thus emerges from the chaos. A good comparison is a sculptor freeing a figure from a rough block of stone, with the text serving as the blueprint.

A common misconception is that the model assembles finished image parts from a database. It does not do that. What is stored are only billions of numerical values that describe patterns and relationships. This is precisely why it fails on details it has rarely seen. Hands with six fingers or illegible text on signs were long typical errors, though they have since become rarer.

Image models in apps, news, and school

Well-known names include Midjourney, Stable Diffusion, DALL·E, and Google Imagen. Many of these can be tried out for free, usually with a limit on the number of images per day. Chatbots such as ChatGPT or Gemini also have an image model built in. You only notice this because a request for an image results in an image appearing.

Less conspicuous are image models in phone apps. The feature that removes an annoying passerby from a vacation photo fills the gap with invented pixels. Portrait filters and the extending of a too-tightly cropped photo also work this way. Here, the model doesn’t generate a whole image but merely fills in missing parts.

In news articles, the term usually comes up in three contexts: copyright lawsuits, fake images surrounding elections, and the billion-dollar valuations of providers. It’s important here to distinguish this from a language model, which generates text rather than images. Systems that process both are called multimodal. And a video model is essentially an image model that additionally ensures that consecutive individual frames match up with each other.