Key-Value Cache

A language model like ChatGPT writes its answer word by word. For every new word, it looks back at everything that has already been written. If it had to recompute this look-back completely from scratch every time, that would be an enormous waste. After all, the old text no longer changes. The key-value cache is the temporary store in which the model keeps the computed results for the old text. For the next word, it simply draws on this store instead of starting over from the beginning.

Why chatbots would be agonizingly slow without it

Without this store, the effort per word grows with the length of the text. By the thousandth word, the model would have to reprocess a thousand earlier words. The response would get slower and slower the longer it becomes. With the cache, on the other hand, the effort per word stays largely constant.

That’s not a small difference. For long texts, the cache easily saves ten to a hundred times the computational effort. This is precisely why a chatbot can write fluently line by line instead of stalling for minutes. Computing time costs money, so the cache also directly lowers a provider’s operating costs.

The price for this is memory space. The cache sits in the fast memory of the graphics card, which is scarce and expensive. In very long conversations, the cache can take up several gigabytes. Sometimes it’s not computing power but exactly this memory hunger that limits how many users a server can serve at the same time.

What keys and values actually are

Modern language models work with a mechanism called attention. For every new word, the model checks which earlier words are currently important. Each word in the text is given two data packages for this purpose: a key and a value. The key is something like a search term used to locate the word. The value is the content the model adopts if it considers this word relevant.

You can picture this like a card index box. At the top of the card is the keyword, at the bottom the content. The new word flips through the keywords and reads on wherever it fits. Once the keys and values of the old words have been computed, they no longer change. That’s exactly why they can be easily stored and reused.

The cache fills up in two phases. First, the model processes the entire input at once and stores a key and value for each word. This phase is called prefill. After that, exactly one new entry is added for each generated word, while all the old ones remain in place. The cache thus grows steadily with the length of the conversation.

Where the KV cache shows up in news and products

Reports about AI data centers often talk about graphics cards with huge amounts of memory. One reason for this is the key-value cache. The stated context window — that is, how many words a model can take in at once — also depends on it. Context windows of a million words are difficult mainly because the associated cache becomes enormous.

Many providers now advertise so-called prompt caching. With this, the cache is preserved across multiple requests for a consistent prior context. Anyone who always sends the same long instruction to their program often then pays only a fraction for it. With some providers, that amounts to roughly a 90 percent discount on the repeated parts.

A common misconception is that the cache makes the model smarter. It doesn’t. The result is computationally identical to what it would be without the cache — it just arrives faster. The key-value cache is pure efficiency, not an additional capability.