KV Router

Large AI text programs like chatbots don’t run on a single machine, but on many servers simultaneously. Someone has to decide which server handles a newly arriving question. This distribution point is generally called a load balancer. A KV router is a particularly clever variant of this. It doesn’t just look at which server currently has little to do, but also which server already “remembers” the conversation so far. That’s because the AI program stores intermediate results in memory for every text it has read. Whoever already has the matching intermediate results finishes the answer much faster.

Why a second server does the same work twice

When you write the tenth message in a chat, the model doesn’t just read that message. It reprocesses the entire conversation so far, because only then does your question make sense at all. This preprocessing of the entire text is called prefill and is the most expensive part of a request. In a long conversation or an uploaded PDF, that can amount to tens of thousands of words.

That’s exactly why every server caches its intermediate results. But if your next message happens to land on a different server, that cache is useless. The new server starts from scratch and recomputes the entire conversation history. For you, this means a noticeably longer wait until the first word of the answer appears. For the operator, it means computing costs that could have been avoided.

With millions of users, this adds up enormously. Providers report that cache-aware routing can reduce the time to the first word by a multiple. Pricing reflects this too: several providers charge only a fraction of the normal price for reused text portions. A KV router is therefore not a technical detail, but directly a matter of money and speed.

How the router finds the right server

The name comes from the KV cache. KV stands for Key and Value, two kinds of numerical values that the model generates and retains while reading a text. You can think of them as notes the model makes for every word it has already read. These notes reside in the memory of the graphics card on which the model runs. They are bound to exactly that one server.

The router itself doesn’t perform any computation on the model. Instead, it calculates a kind of fingerprint of the beginning of your text, usually block by block for chunks of a few dozen words each. It then compares these fingerprints against a list that records which server currently has which blocks in stock. The server with the longest match at the start of the text gets the job.

The restriction to the beginning of the text is important. A server can only continue from the point where the text is word-for-word identical up to a certain position. If a word changes in the middle, everything after it becomes worthless. That’s why developers always place fixed instructions and documents at the beginning of the request and put the varying user question at the end. The router must also weigh things up: if the ideal server is already overloaded, an empty server is sometimes the faster choice despite having to recompute everything.

Which products use it

A KV router is never directly visible. You only notice it by the fact that an answer in a long chat starts surprisingly fast. It’s built into almost every major serving framework for AI models, such as the vLLM and SGLang servers or NVIDIA's Dynamo software. Kubernetes, the widely used server management system, has also meanwhile gained its own building blocks for cache-aware routing.

In trade news, the term usually appears in the context of prompt caching and inference costs. When a provider advertises cheaper prices for repeated inputs, there is almost always such a router behind it technically. For companies running their own AI systems, it’s a standard tool. A common mistake is confusing the KV router with the router in a mixture-of-experts model. The latter selects building blocks within the model, whereas the KV router selects an entire server in the data center.