Model Routing

Providers of AI services usually don’t offer just one program, but several. The large ones deliver better answers, but require a lot of computing time and cost a corresponding amount of money. The small ones are fast and cheap, but fail at difficult tasks. Model routing is the technique that automatically assigns each individual request to the appropriate program. An additional program, the router, briefly looks at the question and decides: simple or difficult. Ideally, the user notices nothing of this and only sees an answer.

What providers save in computing costs through this

Every answer from an AI service costs the provider real money, because specialized computer chips have to compute for it. With millions of users per day, this adds up to enormous sums. At the same time, most requests are surprisingly mundane: translations, summaries, spelling corrections. Using a top-tier model for such tasks is wasteful.

This is exactly where routing comes in. If a service hands off 80 percent of its requests to a small model, costs drop drastically. Depending on the application, providers report savings of between half and nine-tenths. For publicly traded technology companies, this is not a minor detail but a direct effect on profit margins.

But it’s not just about money. Small models respond faster, often in under a second. Anyone running a search bar or a customer service chat gains noticeably in speed as a result. In the best case, routing therefore improves both cost and waiting time simultaneously.

How the router assesses the difficulty of a question

The simplest variant works with fixed rules. If the request contains program code or a math problem, it goes to the large model. If it is short and harmless, the small one takes over. Such rules are written by hand by a human, and they are easy to follow.

More sophisticated systems use their own, very small learning program as a router. It was trained with many example requests for which it was known which model had still solved them correctly. From this, the router estimates for a new request whether the small model will likely suffice. This estimate has to happen extremely quickly, otherwise the router eats up the time savings again.

A third variant is escalation. First, the small model always answers, then a control step checks the result. If the answer seems uncertain or contradictory, the question is passed on to the large model. This is more reliable than an upfront estimate, but costs double in case of doubt. Important for distinction: model routing distributes requests among several separate models. With the similarly named Mixture of Experts, by contrast, the router sits within a single model and chooses between its building blocks.

Routing in chatbots, cloud services, and the news

Anyone using a well-known chatbot has very likely already experienced routing. Several providers have done away with selection menus for models and instead built in automatic assignment. With some services, this can be recognized by a note such as “thinking more carefully” or by a suddenly longer waiting time. The system has then internally switched to a stronger model.

In companies, routing often runs through an intermediate layer that addresses several providers simultaneously. The same software can send a request to a model from OpenAI, Google, or an open-source project. This way, the company becomes less dependent on a single supplier and simultaneously reduces its bill.

In the news, the term mostly comes up when providers talk about falling operating costs. But it also comes up in complaints. Users sometimes notice that answers get worse for no apparent reason, and suspect a silent rerouting to a cheaper model behind it. A common misconception, incidentally, is to consider routing a cost-cutting measure at the expense of quality. When well configured, it changes nothing about answer quality; when poorly configured, it certainly does.