Cache Hit (LLM Inference)

When you send a question to an AI text program like ChatGPT, the program first reads in your entire text. In doing so, it calculates intermediate values for every word before even the first word of the answer is produced. This calculation costs time and electricity. Now, many applications repeatedly send off nearly the same text, for example a long instruction at the beginning of every request. A cache hit occurs when the system notices: I have already calculated this beginning before, the intermediate values are still ready. Instead of recalculating, it retrieves the stored values. The counterpart is called a cache miss: then nothing suitable is stored and everything must be calculated from scratch.

What a hit saves in money and waiting time

The most noticeable effect is the waiting time until the first word of the answer. Experts call this Time to First Token. With a very long input text, this waiting time can amount to several seconds. With a cache hit, it often shrinks to a fraction of that, because the expensive first computation step is skipped.

Then there is the price. Providers like OpenAI, Anthropic, or Google charge for their models based on the amount of text, measured in tokens. A token is a text building block, usually a short word or a syllable. For cached tokens, providers charge significantly less, depending on the service, only a tenth to a quarter of the normal price. Anyone running an application with millions of requests per month thereby saves a great deal of money.

That is why the share of cache hits among all requests, the so-called hit rate, is an important metric for developers. A high rate means cheap and fast applications. A low rate often indicates that the requests are poorly structured.

Why only the beginning counts

A language model processes text from left to right. The intermediate value for the twentieth word depends on what was in the first nineteen words. If you change a word at the beginning, all subsequent intermediate values are no longer valid. That is why the cache only works for a contiguous piece from the start of the text, the so-called prefix.

You can imagine it like a recipe that is worked through step by step. The first ten steps are the same for every dish, so you prepare them in advance and keep them chilled. But as soon as someone changes step two, the entire preparation becomes worthless. That is precisely why all providers advise placing unchanging parts at the front and variable parts at the end.

Technically, what gets stored are the values of the so-called KV cache, i.e. blocks of numbers in the graphics card memory. This memory is expensive and limited. That is why entries remain only a few minutes if no one requests them again. A common misconception, by the way, is that the cache stores finished answers. It does not: it only stores the preprocessing of the input, the answer is generated anew each time and can turn out differently.

Where cache hits show up in everyday life

You notice the effect most clearly in a longer chat. Every new question sends the entire previous conversation along again. The old part is unchanged and delivers a cache hit, only your new question has to be freshly calculated. Without this method, long conversations would become ever slower and more expensive.

Coding assistants like GitHub Copilot or Cursor also work this way. They place a project's source code at the beginning of every request. As long as you are only typing in one spot, the rest stays the same and comes from the cache.

In financial and tech news, you usually encounter the term in the context of costs. When a provider advertises sharply reduced prices, this often refers to cached inputs, not the full rate. It is therefore worth taking a look at the price list: there you will find separate lines for new and for cached tokens.