Thinking Budget

Modern chat programs like ChatGPT or Gemini don’t always deliver their answer right away. Some of them first spend a while computing on their own, jotting down notes that the user never actually sees. This silent preliminary work costs computing time, and computing time costs money. A thinking budget is the upper limit set for this. Essentially, you tell the program: You may do at most this much preliminary work, then you must answer. In practice, this limit is usually specified in word fragments, that is, in the amount of text the program is allowed to produce internally.

Why thinking suddenly costs money

Until recently, the rule was: An AI model is trained once, at great size and expense, and after that every answer is roughly equally costly to produce. Since 2024, that has shifted. Models are now deliberately trained to write out intermediate steps for themselves before answering. For math, programming tasks, or nested questions, this makes the answers considerably better. But the price for this is no longer paid just once during training — it’s paid with every single request.

That’s precisely why a dial is needed. For a question like the capital of Peru, long deliberation is pure waste. For a proof in geometry, it can make the difference between right and wrong. Without an upper limit, a model would keep computing for minutes even on trivial questions, racking up costs nobody needs.

For companies that build such models into their own products, the budget is therefore a business-economics variable. It helps decide whether a customer-service assistant costs cents or euros per conversation. A budget set too low saves money but makes the product worse. This trade-off is a central topic whenever providers discuss the profitability of AI services.

From dial to bill

Technically, the model writes its intermediate steps as ordinary text, just into an internal area. This text is measured in tokens, i.e., small chunks of text roughly four characters long. A thinking budget of 8,000 tokens allows the model around 6,000 words of preliminary work. Once it hits the limit, it cuts off its train of thought and formulates an answer with whatever it has.

Among the major providers, the budget can be set in various ways. Google's Gemini allows a specific token count as well as the value zero, which switches off thinking entirely. Anthropic and OpenAI tend to work with tiers such as low, medium, and high. Some models by now estimate for themselves how difficult a question is and pick the tier automatically.

There’s an important misconception to address here: more budget doesn’t automatically mean better answers. Beyond a certain point, additional computing time yields hardly any improvement. For simple questions, prolonged deliberation can even make the answer worse, because the model gets lost in tangents. The budget is an upper limit, not a quality guarantee.

Where the dial shows up in everyday use

As a user, you rarely see the budget as an actual number. But you do encounter it in disguised form. When you choose between a fast and a thorough mode in a chat app, you are, at heart, adjusting exactly this dial. Indicators like “Thinking…” while you wait also signal that a budget is currently being used up.

For developers, on the other hand, it becomes directly visible. Anyone who calls a model through a programming interface passes the budget along as a parameter and pays for the tokens consumed. That’s why thinking budgets regularly show up in pricing tables and product announcements. When a provider announces that its new model achieves the same performance with half as much thinking, this is exactly the cost being referred to.