Context Compaction

A chat program like ChatGPT cannot remember an unlimited amount. Before every response, the program is shown the entire conversation history up to that point once again. This text that is shown has a fixed upper limit, measured in text chunks. In a very long conversation, the space eventually runs out. Context compaction is the solution to this: the system condenses the older part of the conversation into a short text and continues working with this summary from then on. This greatly reduces the space required, and the conversation can continue.

Why conversations would otherwise simply break off

The amount of text a model can read at once is called the context window. It is measured in tokens, i.e. word chunks. A token roughly corresponds to a syllable or a short word. Typical models today process between 100,000 and one million tokens. That sounds like a lot, but it gets used up quickly.

This happens especially fast with coding assistants. Anyone who has a model work on a real software project is constantly sending file contents, error messages, and test results back and forth. After one or two hours of work, the window is full. Without compaction, one would have to start a new conversation and explain everything all over again.

On top of that, there is a cost argument. With most providers, you pay per token processed. A full context window on every single response is therefore expensive and also slow. A summary of 2,000 tokens instead of 150,000 tokens of conversation history drastically reduces both. Compaction is thus not merely a stopgap solution, but also a cost-saving measure.

Summarizing without discarding what matters

The usual process is simple. The system monitors how full the context window is. If it exceeds a threshold, say 80 percent, compaction is triggered. The model is then instructed to summarize the prior history itself. This summary replaces the old messages, while the last few messages usually remain in their original form.

What matters is what goes into the summary. Good systems give the model precise instructions for this. Typically recorded are: the user’s goal, decisions made, files worked on, errors encountered, and the next planned step. Pleasantries and intermediate steps that led nowhere are dropped. It is similar to a handover report during a shift change in a hospital: not every word is passed on, but everything that matters for the next shift is.

The catch is obvious. Every summary loses information, and one cannot know in advance which details will be needed later. A casually mentioned wish can disappear after compaction. For this reason, the technique is often combined with external memory: important things are written to a file that the model can retrieve again when needed. Compaction must also be distinguished from finetuning, in which a model is permanently retrained. Compaction does not change the model at all, only the text that is presented to it.

How to recognize compaction in use

The process is most visible in coding tools such as Claude Code, Cursor, or GitHub Copilot. There, in the middle of a long session, a notice appears stating that the history has been compressed. Some tools additionally display a fill level of the context window. Compaction can also often be triggered manually with a command before starting a larger new subtask.

Something similar also happens in the background in ordinary chat interfaces, just without any announcement. If an assistant suddenly forgets a detail from the beginning after a hundred messages, compaction was usually involved. This is not a bug in the program, but the result of a deliberate trade-off between space and completeness.

In trade news, the term mainly comes up in connection with AI agents, i.e. systems that work through tasks autonomously over hours. For such agents, compaction is a basic requirement, since their tasks would otherwise fail at the context limit. Providers are therefore now explicitly advertising how well their compaction works.