Routing

When you ask an AI a question, there is usually more than one way to answer it. Some questions are simple: “What time is it in Tokyo?” Others demand lengthy reasoning, such as a mathematical proof. Routing is the automatic decision about which path is taken. A small auxiliary program briefly looks at the request and forwards it to the appropriate place. The name comes from the English word for “choosing a route,” like a package being placed on the correct conveyor belt at a sorting facility.

Why the detour through a dispatcher pays off

Large AI models are expensive to operate. Every answer costs computing time on specialized chips, and that computing time costs money and electricity. Anyone running a system with millions of users notices the difference immediately. If a powerful model explains how many legs a spider has, that’s a waste. A much smaller model can answer that question just as well, often ten times cheaper.

The second reason is speed. Large models take longer before the first letter appears. In a chat follow-up, a delay of several seconds is noticeably unpleasant. Routing ensures that simple requests are answered almost instantly. Only for difficult tasks does one deliberately accept the wait.

Third, routing enables specialization. One model may be especially good at programming, another especially good at summarizing texts. Instead of building a single all-purpose model, several are combined and the dispatcher chooses among them. Providers save costs this way without any noticeable loss in quality.

How the dispatcher makes its decision

The dispatcher is itself a small program, often a tiny AI model. It reads the request and estimates two things: How difficult is the task? And which model is likely to solve it well enough? This assessment takes milliseconds and costs almost nothing. Only after that does the actual work begin.

Such a dispatcher is trained using examples. Many requests are processed by all available models, and the results are compared. From this, the dispatcher learns patterns: for questions of this type, the small model is sufficient; for that type, it is not. Some systems also work with fixed rules, for example based on text length or detected programming language.

There is routing on two levels that are easily confused. In one, a choice is made among several independent models. In the other, a router decides within a single model which of its components should compute. This second form is called Mixture of Experts and is now built into almost all large models. Both share the same basic idea: don’t switch everything on, only what’s necessary.

Routing in chatbots and in the price list

When you see a toggle in a chatbot like “fast” or “think thoroughly,” that’s manual routing. With many providers, this same decision now happens automatically in the background. You only notice it because some answers appear instantly and others only after a pause for thought. This very automation was a point of contention when providers introduced it without warning.

In business news, routing comes up when costs are discussed. Companies that build AI into their products pay per request. Good routing can cut this bill in half without customers noticing any difference. That’s why there are dedicated providers whose only product is smart distribution across third-party models.

A common misconception: routing doesn’t make a system smarter. It only distributes existing capabilities more cleverly. If the dispatcher misjudges a task, a tricky question ends up with the weaker model. The result is then worse than it should have been. Good routing is therefore always measured by both: costs saved and quality maintained.