Looped Transformer

A Looped Transformer is an AI language model with an unusual property: its inner building blocks — the layers that process text — are not run through once, but multiple times, like a loop. This sounds simple, but it has significant consequences. A normal language model stacks many different layers on top of each other, each with its own settings. A Looped Transformer instead takes a smaller group of layers and runs the data through that same group again and again. This makes the model more compact in construction, while it still reasons in many steps.

Fewer building blocks, more depth

The size of an AI model is often measured in parameters — these are the numerical values that the model learns during training and that store its knowledge. The more parameters, the more memory and computing power the model requires. For a given processing depth, a Looped Transformer has significantly fewer parameters than an equivalent normal model, because it reuses the same layers multiple times instead of adding new ones.

This is not merely a cost-cutting measure at the expense of quality. Studies show that Looped Transformers perform similarly well on certain tasks, such as logical reasoning, compared to classic models with more parameters. The reason: by repeatedly passing through the same layers, the model can refine step by step — almost like a person who reconsiders a difficult task multiple times before answering.

How the loop deepens reasoning

In a classic transformer — the basic building block behind models like GPT — the text data is passed through a fixed sequence of layers, once from top to bottom. Each layer slightly further alters the internal representation of the text. At the end, there is an output. The depth of reasoning depends directly on how many different layers are built in.

A Looped Transformer separates these two things: depth and parameter count. It takes a block of, say, eight layers and runs the data through this block five times — this corresponds to the computational power of 40 layers, but with only the memory requirement of eight. During each pass, the model knows which round it is in. This information is provided as an additional signal, so that the layers can adjust their behavior depending on the pass.

A common misconception is to confuse this method with simply repeating the same computation — that would indeed accomplish nothing. The difference lies in the fact that the output of one pass serves as the input of the next. The model thus actually builds on its own intermediate results and refines them step by step.

Looped Transformers in current models and research

The concept appears mainly in research on efficient models — that is, models that achieve as much as possible with limited resources. Microsoft Research, for example, has investigated Looped Transformers as a basis for parts of the Phi model family, a series of compact language models that, despite their small size, deliver surprisingly strong results.

In practice, this is especially relevant for devices with limited memory: smartphones, tablets, or embedded systems that don’t have a data center behind them. A model that fits 40 layers' worth of computational work into 8 stored layers can be more easily ported to such devices.

In the research literature, Looped Transformers are also used as an argument that AI models could fundamentally benefit more from more computing time — regardless of how many parameters they have. This touches on a larger topic in current AI research: the question of whether it makes more sense to build models wider (more parameters) or deeper (more computational steps).