Context Rot

Programs like ChatGPT can only take a limited amount of text into account at once. This amount is called the context window: everything the program currently has in view during a conversation. Manufacturers state large numbers for this, such as a million words. Context Rot refers to the observation that response quality declines well before this limit is reached. The text technically fits, but the program uses it increasingly poorly. Literally, the term means something like “context decay” – the input doesn’t actually rot, it just gets processed increasingly unreliably.

Why large context windows promise less than they sound

The size of the context window has become a marketing argument. Providers compare themselves using figures like 128,000, 200,000, or a million tokens. A token is a unit of text, roughly a short word or a word syllable. Context Rot shows: this number is an upper limit, not a performance guarantee. It states what fits in, not what is reliably understood.

For businesses, this is a practical problem. Many companies want to have entire collections of contracts, manuals, or quarterly reports analyzed in one go. If the model overlooks an important clause in the middle of the document, errors arise that nobody notices. After all, the model doesn’t say “I’ve lost track.” It keeps responding fluently and confidently, just incorrectly.

Moreover, long context costs money. Billing is usually per token, and computation time increases disproportionately with the amount of text. So you pay more for a result that may be worse than with a shorter, deliberately selected input. This is precisely why Context Rot serves as an argument in expert discussions against the motto “just throw everything in.”

What goes wrong in long inputs

When responding, language models weigh all parts of the input against each other. This mechanism is called attention. With short texts, attention is distributed across few spots and is therefore sharp. With very long texts, that same limited attention has to be spread across thousands of spots. Individual details then barely receive any weight.

A well-studied special case is called “Lost in the Middle.” If you hide a piece of information at the beginning or end of a long document, the model usually finds it reliably. If the same information is located in the middle, the hit rate drops noticeably. It’s similar with humans: from a long lecture, the introduction and the conclusion tend to stick.

A second reason is distraction through similarity. If there are ten passages in the context that all nearly match the question, the model has to pick the right one. The more near-matches there are, the more often it grabs the wrong one. Contradictory material is harmful too: outdated and current versions of a document in the same window lead to mixed-up answers. It’s important to distinguish this from hallucination, meaning freely invented information. With Context Rot, the correct information is present, it’s just not used correctly.

Context Rot in chatbots and coding assistants

The effect is most noticeable in long chats. After two hours of conversation, an assistant suddenly forgets an instruction given at the very beginning. The advice “open a new chat for a new topic” is a direct consequence of Context Rot. Coding assistants that keep an entire software project in context behave similarly: as the number of files grows, so does the number of careless errors.

This is why products frequently employ a technique called retrieval, meaning targeted lookup. Instead of sending along a complete manual, the system first searches for the three matching paragraphs. Another approach is summarization: older parts of a conversation are condensed into a short note. Both pursue the same goal, namely keeping the context small and clean.

In tech news, the term usually comes up on two occasions. First, when a provider announces an even larger context window and experts ask how usable it really is. Second, when benchmarks are published that compare models at 10,000 and at 500,000 tokens. A typical misconception is to treat Context Rot as a bug that gets fixed with an update. It is rather a fundamental characteristic of today’s models, one that is worked around through skillful selection of the input.