Cached Input

When you send a question to an AI text program, the program first reads through your entire text, word by word. This reading takes computing time, and computing time costs money. Now, many applications send the same long text prefix with every request: instructions on how the program should behave, or a document being discussed. This prefix is thus read anew every single time, even though nothing about it has changed. Cached Input is the solution to this: the provider remembers the intermediate result from last time and simply reuses it. Only the genuinely new part of the request still needs to be processed.

Why repeated text prefixes cost so much money

Providers of AI models bill according to text volume. This is counted in tokens, which are text building blocks about four characters long. A word usually consists of one or two tokens. Billing is done separately: once for the text you send in, and once for the text the model outputs.

In modern applications, the input portion is often enormous. A coding assistant in an editor likes to send along several files of source code so the model understands the context. A customer service chatbot resends the entire conversation history plus its behavioral rules with every message. After twenty messages, the ever-identical prefix easily makes up 90 percent of the request. Without a cache, you pay for this prefix twenty times over.

That’s why providers actively advertise the discount. At OpenAI and Anthropic, reused tokens cost, depending on the model, only a quarter to a tenth of the normal price; at Google it’s similar. For a service handling millions of requests per day, this factor determines whether the business is viable at all. As a side effect, latency also decreases, because the first response starts faster.

What exactly sits in the cache

What gets stored is not the text itself, but what the model computed from it while reading. You can picture this like the notes of a person reading a book. The notes are laborious to create, but whoever has them doesn’t need to read the book again. In technical jargon, these notes are called the KV cache, because they consist of two kinds of intermediate values.

For the trick to work, the beginning of the new request must match the old one character for character. This is because the model works strictly from front to back. As soon as one word changes, all notes from that point onward become worthless. A single inserted date at the start of the text thus destroys the entire discount.

This leads to a practical rule: what stays the same belongs at the front, what changes belongs at the back. Instructions and documents first, then, the current user question last. The caches also don’t persist long, often only a few minutes without new usage. A common misconception is that caching lets the model learn something. That is false. It is purely a computational saving; the model itself remains unchanged.

Where the discount shows up in price lists and invoices

You encounter the term most directly in providers' pricing tables. There, alongside input and output, is a third column, usually called “cached input” or “cache read.” Anyone working with a model via a programming interface also sees in the response how many tokens came from the cache. This figure is an important metric for developers.

As a user, you notice the effect indirectly. Coding tools like GitHub Copilot or Cursor, and chatbots with long conversations, would be considerably more expensive without this mechanism. Companies that point an AI system at their own manual also rely on it. The manual then sits fixed at the beginning of every request and remains, practically speaking, permanently in the cache.

In financial news, cache prices come up when discussing the operating costs of AI services. They belong to a group of techniques that don’t improve the models' capabilities but rather make their operation cheaper. Cached Input should be distinguished from a classic response cache. The latter simply outputs the old answer again for an identical question. Cached Input, by contrast, only saves work on reading; the answer itself is recalculated every time.