Model Routing

Providers of AI services usually don't run just one program, but several. Some of them are small and respond in a fraction of a second. Others are very large, take longer to compute, and cost several times as much per answer. Model routing is the technique that decides, for each individual request, which of these programs handles it. So an intermediate layer reads the question, estimates how difficult it is, and forwards it accordingly. The user notices none of this: they type their question into a single input field and get an answer.

Why nobody lets the largest model answer every question

The cost of an AI answer depends heavily on how much computation is involved. Between the smallest and the largest model of a provider, there is often a price factor of ten to a hundred. At the same time, a large share of requests are mundane: translations, summaries, spelling corrections, greetings. For these tasks, a small model delivers practically the same result.

An example makes this tangible. Suppose a service processes ten million requests per day. If three-quarters of them go to a model that is twenty times cheaper, the bill drops by a very large amount without any visible loss of quality. This is precisely why routing is crucial to the economics of many AI products. It is one of the reasons why chatbots can now be offered for free.

Besides money, it's also about time. A small model often answers in under a second, while a large one takes noticeably longer for lengthy answers. Saving on waiting time leads to happier users. And if a provider is currently overloaded, routing can redirect requests to available capacity.

How the decision about the model is made

The simplest variant is fixed rules. If the request contains program code, it goes to a model that is good at programming. If it's shorter than twenty words, the small model takes over. Such rules are written by hand by developers. They are easy to understand, but they get many cases wrong.

More common, therefore, is a learned router. This is itself a small AI program that has been trained to estimate the difficulty of a question. Old requests, along with the information about which model solved them satisfactorily, serve as training material. The router then outputs a kind of difficulty score, and above a certain threshold, the large model is used. The provider can shift this threshold depending on whether quality or cost matters more to them.

A third option is the cascade. Here, the small model always answers first. Afterwards, an evaluation step checks whether the answer looks usable. Only if it doesn't is the question passed on to the large model. This saves money, but in the worst case doubles the waiting time. It's important to distinguish this from Mixture of Experts: there, a router selects building blocks within a single model, whereas in model routing, these are completely separate models.

Routers in chatbots, developer platforms, and the news

Anyone using a major chatbot is very likely already using a router. Instead of a long list of models, providers are increasingly offering an automatic setting. The system then decides for itself whether to answer quickly or to think longer beforehand. This longer thinking is exactly what costs computing time, which is why it doesn't happen for every question.

For companies, there are dedicated services that distribute requests across the models of multiple providers. A company then integrates just one interface and still remains flexible. If a provider goes down or a new model becomes cheaper, only the routing rules need to be changed. The topic comes up in business news when providers talk about falling costs per answer.

A common misconception is the assumption that routing is invisible and harmless. In fact, the same question can be answered with varying quality on different days because a different model was responsible. Users then complain that the AI has suddenly gotten dumber. Often there is no change to the model behind this, but rather a shifted routing threshold.