Token Limit

Programs like ChatGPT don’t read text as words. They first break it down into small building blocks called tokens. A token is sometimes an entire short word, sometimes just a syllable or a punctuation mark. For German text, a rough rule of thumb applies: 100 words yield about 130 to 150 such building blocks. The token limit is now the upper boundary for how many of these building blocks the program can keep in view at the same time. Everything counts together: your question, everything said before in the same conversation, and the answer currently being generated.

Why chatbots eventually forget

The limit is the reason for a behavior that confuses many users. In a long conversation, the program suddenly seems to forget what was discussed at the beginning. It doesn’t really forget – the old part simply no longer fits into what is currently being considered. Older messages drop out the back as soon as new ones are added at the front.

In practice, this means a hard limit for many tasks. A model with 8,000 tokens holds roughly 20 book pages. Anyone wanting to use it to summarize a 300-page business report will fail. That’s why providers now aggressively advertise large limits: current models sit at 128,000 tokens, some at a million and more. That corresponds to several thick novels at once.

A second point is money. Providers bill usage of their models per token, separately for input and output. Anyone who sends along an entire manual with every request pays for it every single time. Maxing out a large limit is technically possible, but rarely the cheapest solution.

What happens during breaking down and computing

When breaking down the text, each building block is assigned a number from a fixed dictionary. Frequent words like “and” are a single token. Rare or compound words fall apart into several pieces. The German word “Donaudampfschifffahrt” becomes a handful of fragments, while the English “and” remains one. That’s why the same content consumes more tokens in German than in English.

The limit itself has a technical reason. When computing, language models compare each building block with every other one in order to recognize relationships. If you double the amount of text, this effort roughly quadruples. An unlimited window would therefore not only be expensive, but computationally infeasible. In addition, the model must have learned during training how to handle long texts – you cannot simply raise the limit afterward.

A common misconception: a large limit does not mean that everything is used equally well. Studies show that models find information at the beginning and end of a long text more reliably than in the middle. The technical term for this is “lost in the middle.” A million tokens of capacity therefore does not guarantee a million tokens of attention.

Where you notice the limit in everyday life

You notice it most clearly when uploading files. A chatbot that rejects a PDF or only comments on the first few pages is hitting its token limit. Cut-off answers also belong to this: for output there is usually a separate, smaller limit of a few thousand tokens. The text then breaks off in the middle of a sentence.

In news about AI, the term usually appears as a selling point. When a company introduces a new model, the context length is almost always in the press release. Larger windows are considered progress because they enable new applications – such as searching through entire contract collections or whole codebases.

Programmers who build AI models into their own software plan firmly around the limit. A common solution is called RAG: instead of sending along an entire document, a preliminary program only retrieves the few matching paragraphs. This saves tokens, saves money, and often delivers better answers than stuffing the window full.