Cache Miss

A computer stores data in various locations. Some of these are very fast, but small and expensive. Others are large and cheap, but slow. So that a computer doesn’t constantly have to wait for slow memory, it stores frequently needed data in a small intermediate storage. This intermediate storage is called a cache. If the computer finds what it’s looking for there, this is called a hit. If it doesn’t find it, that’s a cache miss: the data has to be reloaded from the slower memory.

Why a single access failure costs so much time

The orders of magnitude are extremely different. Accessing a processor’s fastest cache takes a few billionths of a second. Accessing main memory often takes a hundred times longer. If the computer even has to access a hard drive, it becomes noticeably slower still. A single miss sounds harmless. Millions of them per second determine the perceived speed of a program.

That’s why, in practice, one measures the so-called miss rate. It indicates what proportion of all accesses fail. With well-written software, it is often below five percent. If it rises to twenty or thirty percent, the program becomes noticeably sluggish, even though the processor itself isn’t fully utilized at all. It simply spends most of its time waiting for data.

A common misconception is that more cache always solves the problem. That’s only partially true. If a program jumps wildly across huge amounts of data, even a larger intermediate storage helps little. What usually matters more is how the data is arranged and in what order the program retrieves it.

What causes misses

There are roughly three causes. First: data is requested for the very first time and couldn’t have been in the cache yet. Second: the cache is full, older entries have been evicted, and are now needed again. Third: two data blocks compete for the same spot in the cache and displace each other, even though there would still be room elsewhere.

You can imagine this like a desk next to a large archive. On the desk lie the documents you currently need. If a file isn’t there, you have to walk to the archive. Because the way is long, you don’t fetch a single sheet but the entire folder right away. In the same way, upon a miss, a processor doesn’t load a single byte but an entire block of neighboring data.

This leads to an important rule for programmers: data that is needed together should also be located close together. In addition, modern processors try to predict which data will be needed next and load it preemptively. This prefetching significantly lowers the miss rate, but only works with reasonably regular access patterns.

Caches in AI systems and on the web

The principle applies far beyond processors. A browser caches images and fonts of visited pages. On the second visit, the page loads faster because much of it is already available. After clearing the browser cache, every access is a miss, and everything briefly feels slow.

The term appears especially often with language models. When a chatbot responds, it stores intermediate results about the conversation so far. This storage is called a KV cache. If the beginning of a request repeats, the system can draw on it and saves computing time. Providers therefore often charge less for such reused portions than for freshly computed ones.

In company announcements and quarterly results, this topic comes up indirectly. When there’s talk of optimizing operating costs for AI services, it’s often precisely about this: serving as many requests as possible from the cache instead of recomputing them. Every avoided miss is saved energy and saved hardware time.