Context Token

Programs like ChatGPT don’t read text letter by letter, nor word by word. They break it down into small pieces called tokens. A token is usually a syllable, a short word, or part of a word. As a rule of thumb: a hundred English words yield roughly 130 to 150 tokens. Context tokens are precisely the tokens that the program currently has in front of it at the same time: your question, everything preceding it in the conversation, and the answer it is currently writing. All of this together must fit into a fixed window called the context window.

The hard limit of memory

The context window is the only form of memory such a program has during a conversation. Whatever is not in it simply does not exist for the answer. That’s why a chatbot sometimes seems forgetful: in a long conversation, the oldest messages fall out of the back of the window. The program hasn’t forgotten them—it simply no longer sees them.

The size of this window is therefore one of the most important figures for any new model. Early versions of ChatGPT worked with around 4,000 tokens, roughly eight pages of text. Current models manage 128,000 to over a million tokens. A million tokens roughly corresponds to several thick novels. This makes it possible to read in an entire piece of legislation or a complete code repository at once.

Context tokens are also the computing unit used for billing. Providers charge a price per million tokens, separated by input and output. Anyone building an AI tool into an app therefore calculates in tokens, not in questions or minutes. Long conversations become disproportionately expensive as a result, because with every new question the entire previous history is sent along again.

What happens during splitting and holding together

First, a tokenizer breaks down the text according to a fixed list of common character sequences. Frequent words like “and” become a single token. Rare words like “Donaudampfschifffahrt” split into several parts. English texts therefore need fewer tokens than, say, German ones, because the training lists are strongly geared toward English. Each token is then assigned a number and internally converted into a long sequence of numbers.

When responding, the model compares each token with all the others in the window. This is how it recognizes that a “she” refers to a person from the third sentence. This comparison is the reason for the limit: the computational effort increases roughly with the square of the number of tokens. Twice as much context roughly means four times as much work. That’s why large windows are technically expensive and only became affordable through special tricks.

A common misconception: a large context window does not mean the model uses everything in it equally well. In tests, models reliably find information at the beginning and end, while they more often overlook details in the middle. Experts call this “Lost in the Middle.” More context is therefore not automatically better.

Tokens in price lists and product announcements

With every model announcement, the window size appears in the headline. Figures like “200K context” mean 200,000 tokens. Stock market and tech news also pick up on this number, because it shows which tasks a provider can even handle. A model that reads an entire annual report in one go opens up different business fields than one with eight pages of leeway.

In everyday use, you notice tokens through error messages like “message too long” or by the fact that a chatbot ignores older agreements. Some services automatically summarize older parts of a conversation to save space. Anyone working directly with programming interfaces can see in the billing exactly how many input and output tokens each request cost. Context should be distinguished here from training: training knowledge is permanently embedded in the model, while context tokens disappear again after the conversation.