Cache Reads

When you ask a chat program like ChatGPT a question, you send text to a computer in a data center. That computer first has to process the entire text before it can respond. This costs time and electricity. Now, many applications send the same long block of text with every request, such as an instruction set or an uploaded document. A cache read means: the computer has already stored the result of that earlier computation and simply retrieves it. It saves itself the work and delivers an answer faster.

Why providers charge less for it

Anyone using an AI model via a programming interface, meaning in an automated way rather than through a website, pays per processed piece of text. These pieces of text are called tokens; one token corresponds to roughly half a word. Typical prices are a few euros per million tokens. For a cache read, the major providers often charge only ten percent of the normal price. The reason is simple: the expensive part of the computational work has already been done and paid for.

For companies, this is not a minor detail but a tangible cost factor. A customer service bot is presented with the same behavioral rules on every single request, perhaps 5,000 tokens long. With a million requests per month, that adds up to five billion tokens that are actually always the same. With caching, precisely this cost item drops by around ninety percent. That is why cache reads now appear as a separate line item on invoices and in the quarterly reports of AI companies.

The second benefit is speed. The first response arrives noticeably sooner because the model no longer has to churn through thousands of tokens. For coding assistants that keep entire files in view, this makes the difference between smooth work and disruptive waiting.

What exactly gets cached

A language model does not read a text the way a human does, from beginning to end with its eyes. It converts each token into a long list of numbers and calculates how the tokens relate to one another. These intermediate results are called the key-value cache. It is precisely these numbers that are retained, not the text itself. If the same input comes in again, the system loads the finished numbers and continues the computation right where the new part begins.

The order matters. The cache only works from the beginning of the text onward, identical character for character. If you change a single word right at the start, the entire rest becomes worthless and has to be recalculated. Developers therefore deliberately structure their inputs this way: unchanging parts at the top, the actual user question right at the bottom. A common mistake is placing a current date or time at the beginning – doing so destroys the cache with every single request.

The cache is also not valid forever. Typical durations are five to sixty minutes, after which the space is freed up. Some providers charge a small surcharge of about twenty-five percent for creating the cache, known as the cache write. This only pays off once the same block of text is retrieved multiple times afterward.

Where the figure shows up in invoices and reports

In the billing overviews from OpenAI, Anthropic, or Google, cache reads appear as a separate line item alongside normal input and output tokens. Anyone building their own application can see immediately whether the caching is taking effect. A low hit rate is usually a sign of a poorly structured text.

The term also plays a role in news about AI companies. When providers report that their cost per request has fallen, this is often due, alongside better hardware, simply to better caching. As a user, you only notice this indirectly: tools that keep a long document or an entire software project permanently in view have only become affordable in the first place because of this.