Test-Time Compute

An AI system like ChatGPT goes through two quite different phases of its life. First, it is trained over weeks on huge amounts of text until it recognizes patterns in language. After that, it is finished and answers users' questions. Test-Time Compute is the computational work that occurs precisely in this second phase – that is, during the answering itself. The name comes from the fact that researchers used to call this phase the test run, as opposed to the training phase. The central observation behind it: if you simply let a system compute for longer at this moment, its answers become measurably better.

A second lever besides ever-larger models

For years, the AI industry followed a simple rule: more data, more data centers, larger systems. This worked well, but became extremely expensive. A single training run costs the big providers hundreds of millions of dollars. Moreover, starting in 2024, signs mounted that simply scaling up was yielding diminishing additional benefit.

Test-Time Compute opened up a second path. Instead of making the system larger, you give it more thinking time per question. OpenAI demonstrated this in September 2024 with the model o1, which performed dramatically better on math problems than its predecessor. The underlying technique was the same, except the system was allowed to compute significantly longer before answering.

Economically, this shifts costs. Previously, a provider mainly paid once for training. Now, a large share of the cost arises with every single request. This explains why companies like Nvidia keep selling ever-new specialized chips for ongoing operation as well.

Thinking in intermediate steps and multiple attempts

The most common method is called Chain of Thought. Before giving the actual answer, the system writes itself a kind of scratch note: intermediate steps, partial calculations, considerations. Users usually never see these notes. But they help the system avoid carrying an error from the third step all the way through to the final result.

A second method is simple repetition. The system generates five or twenty proposed solutions for the same task. It then selects one of them, for instance the answer that occurred most frequently. For tasks with a clearly verifiable result, such as calculations or program code, this works surprisingly well. Sometimes a second, smaller model checks the proposals and rates them.

An important distinction: the system does not learn anything new in the process. Its knowledge stays exactly the same; only its use becomes more thorough. A common misconception is also that more computing time always helps. For a factual question whose answer the system simply does not know, extended thinking accomplishes nothing. The effect is strongest for tasks involving multiple logical steps.

Visible in the thinking mode of chatbots

Anyone using ChatGPT, Google Gemini, or Claude today can often switch on Test-Time Compute directly. These services offer a mode with names like “Reasoning,” “Thinking,” or “Denken.” The answer then takes maybe thirty seconds instead of two. Often, a running hint text accompanies this, roughly showing what the system is currently working on.

In the news, the term usually comes up in connection with scaling, that is, the question of how AI systems will become better in the future. Analysts discuss whether compute at answer time can take over from the growth of training costs. Chinese providers such as DeepSeek also became known for implementing this approach very cheaply.

For users, this has a practical consequence. In paid tiers, thinking mode costs more or is more limited than the fast mode. For a recipe idea, it’s not worth it. For a difficult physics problem or debugging program code, it often is.