Token Efficiency

Language AI programs like ChatGPT don’t read text as letters or as whole words. They break it down into small building blocks, so-called tokens. A token is usually a syllable, a short word, or a word fragment; “soccer field”, for example, splits into several pieces. Each of these building blocks costs computing time and thus money, whether it appears in the question or in the answer. Token efficiency is the question of how much usable output you get per building block. A system is token-efficient if it arrives at the same or a better result using as few building blocks as possible.

What tokens cost

Providers of AI models charge almost always per token. Typical prices are a few cents per thousand building blocks. That sounds harmless, but it adds up quickly. A company that has millions of customer inquiries answered daily processes billions of tokens per month. Anyone who saves twenty percent tokens there saves twenty percent of the bill.

Tokens also cost time. The model generates its answer building block by building block, one after another. An answer twice as long thus takes roughly twice as long until it’s fully displayed on the screen. For a chat window, that’s the difference between one second and five seconds of waiting time.

A third reason is limited memory. Every model has a window in which only a certain number of tokens fit at the same time. If the text gets longer, older content falls out or has to be shortened. Anyone who formulates sparingly fits more actual content into this window.

How to save tokens

The first lever lies in the input. Many applications send the same long preamble with every request, such as rules for the model’s behavior. Shortening this preamble reduces costs with every single request. Database excerpts given to the model can also be condensed down to the truly relevant paragraphs.

The second lever lies in the output. Models tend to ramble politely and repeat questions. A clear instruction like “answer in a maximum of three sentences” often halves the answer length. Some models “think out loud” before answering, generating many intermediate steps in the process. These intermediate steps improve the result for math and logic, but cost tokens. For simple questions, they are pure waste.

A common misconception: token efficiency is the same as brevity. That’s not true. An overly terse answer that requires a follow-up question ends up consuming more tokens than a good first answer. The tokenization itself also plays a role. German and especially Turkish or Arabic texts require significantly more tokens than English with many models, because the building blocks were calibrated on English text.

Token efficiency in price lists and quarterly figures

Anyone using an AI provider’s programming interface sees their bill broken down directly into tokens, separated by input and output. In the price lists of OpenAI, Google, or Anthropic, the price is listed per million tokens. There you also find so-called caching: repeated text portions are cached and billed more cheaply. That is token efficiency as a product feature.

In news about AI companies, the term mostly appears in connection with margins. A provider whose model needs half as many tokens for the same task can offer lower prices or earn more. When a new model is advertised as “more efficient”, it is almost always about cost per token or tokens per task.

Even as a private user, you feel the effect. Chatbot subscriptions have usage limits based on the amount of text processed. Anyone who copies an entire book into the chat window uses up their quota faster than someone who inserts only the relevant chapter.