Router LLM
A Router LLM is a system that automatically forwards each user request to the appropriate language model. Simple questions go to a small, cheap model, difficult ones to a large, expensive one — this lowers costs without noticeably degrading answer quality.
Large AI language programs like ChatGPT come in various sizes. Small variants respond quickly and cost little to run, but struggle with tricky tasks. Large variants can do more, but require significantly more computing time and money. A Router LLM is the switching point in between. It briefly examines each incoming question and decides which program should answer it. The user notices none of this: they type their question into a single input field and get an answer back.
Why sorting requests pays off
Operating an AI service costs money with every single answer. Experts call this usage phase inference, as opposed to the one-time training of the model. With millions of requests per day, fractions of a cent add up to considerable amounts. The difference between a small and a large model is often a factor of ten to fifty.
The crucial point: most requests aren’t actually difficult. “How do you spell rhythm?” or “Summarize this email in two sentences” can be handled flawlessly even by a small model. Only a fraction of questions genuinely require deep thought, such as a math proof or a lengthy programming task. Anyone who sends everything to the largest model pays for performance that, in most cases, nobody needs.
Studies and vendor figures often cite savings of 50 to 80 percent with almost unchanged quality. Besides money, speed also plays a role. Small models respond faster, which makes the service more pleasant for the user. And less computing time also means less power consumption.
How the switching point decides
The decision must be made extremely quickly, otherwise it eats up the time saved. That’s why the router itself is usually a small, specialized program. It reads the question and estimates how difficult it is. Some routers work with a simple classifier, i.e., a sorting program that divides requests into categories such as “easy,” “medium,” and “hard.”
Such a router is trained with examples. Tens of thousands of real questions are sent to all available models and the answers are compared. From this, the router learns a pattern: for which type of question does the small model deliver the same result as the large one? It is precisely these questions that it later routes downward.
It’s important to distinguish this from a related term. In Mixture of Experts, the router sits inside a single model and distributes work to its internal building blocks. A Router LLM, by contrast, sits in front and chooses between several completely independent models. Another typical misconception is to consider the router infallible. If it misjudges a question, it ends up with the too-weak model and the answer becomes worse.
Router LLMs in chatbots and cloud services
Anyone using an AI chatbot today has very likely already experienced a router. OpenAI introduced an automatic switch between a fast response mode and a longer thinking mode with GPT-5. Other providers, too, rely on the service itself deciding how much effort a question is worth. This sometimes becomes visible through a note like “thinking,” which appears for some questions and not for others.
For companies building AI into their own products, routers have now become standard tools. There are ready-made services that distribute requests among models from different providers. A customer service chat then sends standard questions to a cheap model and forwards complaints to a powerful one.
In financial and tech news, the term usually comes up in connection with costs. When a provider reports having halved its operating costs per request, routing is often behind it. There is criticism too, however: users frequently don’t know which model just answered. For paid subscriptions, this raises the question of whether one is really getting the performance one is paying for.
