KV Cache Compression

When a chatbot writes a response, it generates it word by word. For every new word, it must take into account the entire text generated so far. So that it doesn’t have to recompute everything from scratch each time, it stores intermediate results in a memory. This memory is called the KV cache, short for key-value cache, meaning an intermediate store of keys and values. It keeps growing with every word and occupies expensive space on the graphics card. KV cache compression is the umbrella term for all techniques that make this memory smaller without noticeably degrading the quality of the responses.

What the memory hunger costs in practice

The KV cache is the secret bottleneck of large language models. In long conversations, it can take up more space than the model itself. A medium-sized model quickly needs several gigabytes just for this intermediate memory for a conversation of 100,000 words. And this space is needed per simultaneous user, not once for all of them combined.

This has a direct connection to price. The less memory a conversation occupies, the more conversations fit simultaneously on the same graphics card. Providers convert this into requests per second per card. Halving the cache ideally doubles the throughput. That’s why cache compression is often more worthwhile for operators than a better model.

The marketing claims about context length also depend on this. When a provider advertises a window of one million words, there is almost always a compressed cache behind it. Without such techniques, such lengths would be technically possible but economically pointless.

Forgetting, storing more coarsely, summarizing

The simplest approach is discarding. The model keeps only the most recent words and a few particularly important ones from the beginning, and deletes the rest. This works because many intermediate results hardly matter for the next prediction. You can imagine it like notes from which you strike out everything you’d never look up again anyway. The downside: what’s deleted is gone for good. If someone later asks about a detail from earlier, the answer comes out wrong.

The second approach stores more coarsely instead of less. The intermediate results are numbers, and numbers can be stored with varying precision. Instead of sixteen digits, you store four or eight. This technique is called quantization and shrinks the cache to a quarter, depending on the setting. The loss in quality is usually small, because small rounding errors tend to average out again in the overall calculation.

The third approach builds the compression directly into the model. With multi-query attention or grouped-query attention, multiple computation paths share the same cache entry instead of each creating its own. A newer technique called multi-head latent attention stores the intermediate results in a heavily compressed form and reconstructs them as needed. Such architectures have to be trained this way from the start, but that’s where the effect is greatest.

Where the term appears in reports

KV cache compression is most often encountered in technical reports about new models. When DeepSeek introduced its models in 2024, the compressed storage of intermediate results was a key reason for the low operating costs. Reports about sudden price cuts per request also often have such optimizations as the cause, rather than cheaper hardware.

As a user, you notice the effect indirectly. If a chatbot confuses details from the beginning in very long conversations, an aggressively trimmed cache may be behind it. A common misconception is that the model forgot something the way a human would. In fact, the memory was actively deleted to save space.

The term should be distinguished from model compression. There, the model itself is made smaller, meaning its learned parameters. With KV cache compression, the model remains unchanged; only its working memory during a conversation shrinks. The two are often combined, but they target different bottlenecks.