Heavily Compressed Attention

Modern language programs like ChatGPT read a text word by word. So that they understand the context, the program looks back at all previous words each time it encounters a new one. This looking back is called attention. The problem: the longer the text, the more has to be looked back at, and the effort grows faster than the text length itself. Heavily Compressed Attention is the umbrella term for techniques that drastically condense this look-back. Instead of keeping every single earlier word stored exactly, the program only stores condensed summaries of it.

Why long texts would otherwise become unaffordable

With classic attention, the effort does not double when the text doubles in length. It quadruples. At ten times the length, it’s a hundred times as much. Experts call this quadratic growth. A text of 100,000 words therefore costs not a hundred times but ten thousand times as much computing time as a text of 1,000 words.

Then there’s the memory requirement. While a model is generating a response, it keeps intermediate results ready for every word read so far, the so-called cache. This intermediate memory sits in the graphics card memory, which is expensive and scarce. With very long inputs, this cache alone fills up several gigabytes. This is precisely why chatbots were long limited to a context of just a few thousand words.

Heavily Compressed Attention shifts this limit considerably. Applications that were previously impossible become practical as a result: analyzing an entire legal code, searching through a complete codebase, keeping track of a months-long chat log. Without compression, this would simply be too expensive for most providers.

From word-for-word minutes to summary minutes

A comparison helps here. Imagine you had to take minutes of an eight-hour conference. The exact variant would be a verbatim transcript, containing every sentence. Lossless, but unwieldy. The compressed variant is a summary record with the key statements. Shorter, faster to search through, but details get lost. This is exactly the trade-off that Heavily Compressed Attention makes.

Technically, there are several ways to get there. In one approach, the model combines groups of words into a single collective point, often several dozen words into one. In another approach, it discards entries from the cache that it hasn’t needed for a long time. Yet other methods store the numbers in the cache more coarsely, for instance with four instead of sixteen digits of precision.

Two resolutions are often combined. The last few hundred words remain uncompressed, because they are usually the most important. Everything before that gets heavily condensed. It’s important to distinguish this from a related term: with quantization, the model itself is stored coarsely, whereas here only its memory of the ongoing text is affected.

Context windows in products and marketing promises

In product announcements, you encounter the result of this technique, not its name. When a provider advertises a context window of a million tokens, some form of attention compression is almost always behind it. A token here is a word or syllable unit. A million tokens correspond roughly to several thousand book pages.

A common misconception is that a large context window means perfect recall. That’s not true. Tests show that compressed models sometimes overlook individual details from the middle of long texts. Experts speak of the needle-in-a-haystack problem. Anyone relying on an exact figure from page 300 should therefore double-check it.

The topic also comes up in stock market news. Falling costs per query are a central argument when providers talk about their profit margins. Compression methods lower precisely these costs. For you as a user, the effect simply shows up in the fact that you can upload an entire PDF instead of having to break it into chunks.