Short Context

Programs like ChatGPT can only ever take in a limited amount of text at once. This amount is called the context window: everything the program has in view simultaneously for its current response. We speak of Short Context when this window is small, roughly between about 2,000 and 16,000 text units. For comparison: that’s roughly five to thirty book pages. Anything that no longer fits simply drops out of the response. The program doesn’t notice this and doesn’t point it out either – it simply answers based on what’s still there.

What a small window costs and what it saves

A small window is, first of all, a limitation. A model with 4,000 units can’t review an entire contract or summarize a novel. Even a longer conversation becomes a problem: at some point the beginning slides out of the window. Then the model forgets your name or what the conversation was originally about. Users often take this for a bug, but it’s actually a hard technical limit.

Conversely, Short Context has concrete advantages. The computational effort for processing increases disproportionately with text length. Twice as much text therefore means significantly more than twice the computing time. Whoever runs a model with short context pays noticeably less per request and gets an answer faster. For many tasks, that’s entirely sufficient: rephrasing an email or writing a product description doesn’t need a window for a thousand pages.

There’s also a point that’s often overlooked. Even models with a huge context window don’t work equally reliably across their full length. Information in the middle of a very long text is overlooked more often than information at the beginning or end. A short, well-chosen context can therefore sometimes deliver better results than a long, crammed one.

How developers work around the limit

Before processing, text is broken down into small units called tokens. A token is roughly a short word or part of a word; in English, 1,000 tokens correspond to roughly 750 words. Window size is always given in tokens. Importantly: the question and the answer share this budget. Anyone who fills 3,900 of 4,000 tokens with their question will only get a very short answer.

To work with large documents anyway, people use chunking: the text is cut into pieces and sent through the model individually. The partial results are then combined. It works similarly for chat histories – older messages get condensed into a short note instead of being carried along verbatim.

The most widespread solution is called RAG, short for Retrieval Augmented Generation. Here, a separate system first searches out the matching passages from a database. Only these few paragraphs end up in the window. The model doesn’t need to know the entire manual, just the three relevant pages. This technique is also popular because it works with small, and therefore cheap, models.

Short Context in products and headlines

In a model’s technical specifications, you’ll usually find the window size listed right next to the name. Older models were at 4,000 or 8,000 tokens, while today’s top models advertise a million and more. When a provider announces a larger window in a press release, this is exactly the value in question. Large windows, however, cost more per request – which is why almost all providers also offer smaller, short-context variants in parallel.

In everyday use, you mostly encounter Short Context as an error message. When a chatbot says your text is too long, or when an upload gets rejected, the window is the cause. AI features that run directly on a phone or laptop also almost always work with short context, because little memory is available there.

A common misconception: context window and memory are not the same thing. The window is just the workspace for the current request and gets cleared afterward. When an assistant remembers your preferences over weeks, it stores them outside the model and pushes them back into the window when needed.