Token Budget

Programs like ChatGPT don’t read and write text as a string of letters. They break it down into small building blocks called tokens. A token is usually a syllable, a short word, or a punctuation mark. For English texts, a rough rule of thumb is: 100 words are about 130 to 150 tokens. A token budget defines how many such building blocks may be used at most for a task. This limit applies both to what is sent to the program and to what it writes back.

Why every request has a price

Providers of AI models almost always bill per token, not per request. A typical price list states an amount per one million input tokens and a higher amount per one million output tokens. Anyone running an app with a hundred thousand users notices immediately whether each answer is 200 or 2000 tokens long. A factor of ten translates one-to-one onto the bill.

Besides money, it’s about time. A model generates its answer token by token, that is, sequentially. An answer twice as long takes roughly twice as long. A tight budget ensures that an application feels smooth rather than sluggish.

And there is a hard technical limit. Every model has a context window, that is, a maximum amount of tokens it can take in at once. If a request doesn’t fit, it gets cut off or rejected. The token budget is the practical answer to this limit: you plan in advance how much space each part gets.

How a budget gets divided up

Think of a suitcase with a fixed volume. Everything that’s supposed to go along has to fit inside: the system instruction that states how the model should behave, the conversation history so far, any attached documents, the current question – and space for the answer. Developers therefore reserve fixed shares. One example: 500 tokens for the instruction, 4000 for documents, 2000 for the history, 1000 for the answer.

When things get tight, something has to give. In long chats, the beginning of the conversation is often dropped, or it gets replaced by a short summary. For documents, an upstream system searches out only the most relevant sections instead of sending everything along. Both are cost-saving measures, and both have side effects: the model can forget details it no longer sees.

A common misconception is that a token is a word. That’s roughly true for short English words, but not for German. A long compound word like Donaudampfschifffahrt breaks down into several tokens. Spaces, special characters, and program code, too, cost proportionally more than one would expect. Anyone calculating their budget should therefore measure rather than estimate.

Token budgets in products and headlines

In the pricing tables of OpenAI, Google, or Anthropic, tokens show up directly. Even free offerings work with budgets, just invisibly: when a chatbot asks you after a while to start a new conversation, the context window is usually full. Some coding tools even display how much space is still left.

In the news, this topic often appears as a numbers race. Reports about models with a context window of one million tokens mean that an entire book fits in at once. But that doesn’t solve the cost problem, because a full window is also an expensive window. Providers therefore also advertise discounts for repeated inputs at the same time.

For yourself, one thing matters most: a precise, short question is cheaper and usually also better than a long, disorganized block of text. Anyone who uses models professionally plans token budgets much like a travel budget – with fixed line items and a reserve for the unexpected.