Hybrid Inference Orchestrator

When you ask a chat program a question, some computer has to answer that question. Usually there are several programs to choose from for this: small ones that answer quickly and cheaply, and large ones that can do more but need a lot of computing power. Some run directly on your phone, others in a data center, that is, a hall full of computers. A Hybrid Inference Orchestrator is the switching point in between. It looks at every request and decides who handles it. “Hybrid” here means: device and data center are used in combination, not either-or.

Why big providers don’t send everything to the largest model

Every answer from an AI system costs electricity and computing time. Experts call this phase of use inference, as opposed to training, during which the model first learns. Training happens once, inference happens millions of times over. That’s why it determines the ongoing costs of a service.

A large share of requests is mundane. “How do you spell rhythm?” doesn’t need a model with hundreds of billions of parameters. A small model can handle that for a fraction of the cost. If an orchestrator intercepts about 70 percent of such requests, the bill drops significantly. That is exactly why providers like OpenAI, Google, or Apple build in such switches.

But it’s not just about money. A model on the device answers without an internet connection and sends your data nowhere. That’s a real advantage when dictating a private message. The orchestrator can take such rules into account: some tasks fundamentally never leave the device.

How the switch decides

First, the orchestrator estimates the difficulty of the request. To do this, it often uses a very small model itself, a so-called router. The router receives the question and outputs an assessment of which target model should be sufficient. It is trained with many example requests for which it is known which model was good enough.

Then hard constraints come into play. Is the device in battery-saving mode? Is the connection poor? Is the large model currently overloaded? A good orchestrator knows the wait times and prices of all available models. It chooses the cheapest option that still achieves the required quality.

Many systems additionally work with escalation. The small model answers first, and a check estimates whether the answer is usable. If it seems uncertain, the request goes to the large model once more. A typical mistake is confusing the orchestrator with Mixture of Experts. In Mixture of Experts, a model distributes the work internally among its own building blocks. The orchestrator stands outside and distributes work among several separate, fully independent models.

Where this mediation shows up in everyday life

Voice assistants on smartphones make this principle visible. Simple commands like “set a timer for ten minutes” are handled by the device itself, in a fraction of a second. A long question about a text, on the other hand, is sent to a server. Apple describes this openly for its on-device features, and Google proceeds similarly with its smaller model variants.

The switch is also present in chat services. When a provider writes that the system “automatically selects the appropriate model,” this is exactly the kind of orchestrator meant. For companies, there are also dedicated products for this, often under names like Model Router or AI Gateway. They sit between the company’s own application and the models of various providers.

In financial and tech news, such systems usually come up in connection with costs. When a company reports that it has halved its inference costs, new routing is often behind it. This is relevant for investors because it can dampen demand for data center capacity and strengthen the importance of small models.