Token Routing

A language model first breaks down every text into small chunks. These chunks are called tokens, and in most cases a token is a short word or a syllable. Large models consist of many interchangeable computational building blocks arranged in parallel next to one another. With token routing, the model decides anew for each individual chunk which of these building blocks are allowed to process it. All remaining building blocks stay idle for that chunk. This saves computing time, because the entire model is never used for the computation.

One can picture this like a mailroom in a large company. Every letter is briefly examined and forwarded to the appropriate department. No one would think of sending every letter to all departments. Inside the model, this exact sorting work is carried out by a small auxiliary network, the router.

Why models can stay large and still remain affordable

A model’s capabilities depend heavily on how many learned numerical values it possesses. These values are called parameters, and more parameters usually mean better answers. Traditionally, however, more parameters cost more computing time for every single answer. Token routing decouples these two quantities from one another. The model can possess a very large number of parameters while using only a fraction of them per token.

The effect is considerable. Some models with several hundred billion parameters activate only about thirty billion of them per token. As a result, they respond as quickly as a significantly smaller model. This is precisely why almost all very large models have been built according to this principle in recent years.

For providers, this is above all a matter of cost. Every user request costs electricity and computing time on expensive graphics chips. Anyone who, at the same quality level, only has to run a fifth of the computations massively lowers their ongoing costs. That is why quarterly reports and model announcements often cite the number of active parameters, not just the total count.

How the router makes its decision

The router is itself a very small neural network. It receives the current numerical representation of a token and outputs a score for each building block. Then the one or two highest-scoring building blocks are selected. Only these compute their result, which is subsequently combined in a weighted manner.

It is important to note: no one manually specifies what a given building block is responsible for. The router learns its sorting rules together with the rest of the model during training. What specialization ultimately emerges is often difficult to pin down. Some building blocks respond to program code, others to punctuation marks or to a particular language.

A well-known problem is uneven utilization. If the router sends almost everything to the same two building blocks, the others remain unused. That is why an additional incentive is built into training that rewards an even distribution. In addition, every building block has a capacity limit. Once it is reached, further tokens are sent elsewhere or, in extreme cases, simply passed through unchanged.

From language models to chip utilization

Anyone who reads the term Mixture of Experts in trade news is, indirectly, reading about token routing. Mixture of Experts, often abbreviated as MoE, is the architecture with the many building blocks. Token routing is the mechanism that makes them usable in the first place. Models from Mistral, DeepSeek, Google, and other providers operate according to this principle.

In everyday use, none of this is directly noticeable. One types a question into a chatbot and receives an answer. It becomes noticeable only indirectly, for example through falling prices per unit of processed text or through faster responses even as models grow larger at the same time.

A common misconception is the idea that routing also saves memory. That is not true. All building blocks must reside in the graphics chips' memory at the same time, including the ones currently idle. Only computing time is saved. That is why such models still require a great deal of expensive hardware, and this is precisely one reason for the persistently high demand for data center chips.