DiT Architecture

Many programs that generate images work according to a simple basic principle. They start with a surface of pure random noise, essentially a flicker like an old television without reception. Then they remove this noise in many small steps until an image remains. For each of these steps, the program needs a computational engine that estimates which part of the flicker needs to go. The DiT architecture is a specific blueprint for exactly this computational engine. It uses the same technique found in chat programs like ChatGPT for this: the so-called Transformer.

Why image models changed their engine

Previously, almost all image generators used a different blueprint, the U-Net. This computational engine was specifically designed for images and worked well. But it had a drawback: it scaled poorly. Anyone who built a U-Net twice as large usually didn’t get images twice as good.

With Transformers, it’s different. From language research, we’ve known for years: more computing power and more data reliably lead to better results with Transformers. This relationship is called a scaling law. The DiT paper from 2022 showed that this relationship also holds for images when the U-Net is replaced with a Transformer.

This changed the industry. Anyone wanting a better image model no longer needs to search for a clever new blueprint. They can take the same blueprint and simply pour more compute time into it. This is exactly what the major video and image models of recent years are based on.

From image to tiles and back

A Transformer cannot read images. It processes lists of elements; for text, these are word fragments. So an image must first be converted into such a list. To do this, it is broken down into small square tiles, roughly 2 by 2 pixels in size. Each tile becomes an element of the list. An image thus becomes a sequence of several hundred or thousand tiles.

Now the Transformer’s central capability comes into play: attention. It ensures that every tile can look at every other tile. A tile at the left edge of the image thereby knows what’s happening at the top right. This helps with images that have a coherent structure, such as a face with two matching eyes.

The model also receives two additional pieces of information: which denoising step is currently underway, and what the user wants to see. This information is not fed in as separate tiles. Instead, it steers the computational steps internally. Furthermore, almost all of today’s DiT models don’t operate on actual pixels, but on a heavily downscaled intermediate form of the image. This saves an enormous amount of computing time.

DiT in Sora, Stable Diffusion 3, and other products

The blueprint became best known through Sora, OpenAI’s video model. There, OpenAI described how a video is broken down into space-time tiles. This is the DiT idea, extended to include the time axis. Stable Diffusion 3, Flux, and several Chinese video models are also built on this family.

In news coverage, the term is usually read as the explanation for why a new model looks so much better than its predecessor. An important distinction here: DiT is not a finished product and not a company. It is a blueprint that many providers freely use and adapt.

A common misconception is that DiT replaces the diffusion process. That’s not true. The step-by-step denoising remains exactly as before. Only the computational core that executes each step has been swapped out. In practice, this is felt mainly in the price: Transformers require many graphics cards, which is why video generators remain expensive to operate to this day.