Sparse MoE
Sparse MoE is a design for large AI models in which only a small part of the model computes per request while the large remainder stays idle. This allows models to store a great deal of knowledge without every answer becoming correspondingly expensive.
Modern language programs like ChatGPT consist of enormous computational networks with billions of adjustable numerical values. These numerical values store everything the program has absorbed from texts during training. Normally, all these values are computed for every single question. Sparse MoE breaks with this rule: the network is broken down into many small sub-networks, and for each question only two or three of them work. “Sparse” means exactly that: only a sparse fraction of the program is active at any one time. The English abbreviation MoE stands for “Mixture of Experts” — the sub-networks are called experts.
Large models at the price of small ones
For AI models there is an uncomfortable rule of thumb: more storage slots for learned knowledge mean more capability, but also more computational effort. Computational effort costs electricity, time, and expensive specialized chips. Anyone running a model for millions of users pays this price with every single answer. Sparse MoE separates these two quantities from one another: the model may be huge, while the computation per answer stays small.
An example makes this tangible. A model with 600 billion storage slots can be built so that only about 30 billion of them compute per question. The effort then corresponds to a model twenty times smaller. However, the knowledge of the other experts is not lost — it is called upon whenever matching questions arise.
This is why almost all very large models of recent years are built this way. Several freely available models from China also use the technique. It was an important reason why the cost per answer has dropped significantly.
The router and the question of who answers
The selection of experts is made by a small auxiliary network, the router. For each individual word, it calculates scores for all experts and sends the word to the best two or four. Their results are then combined in a weighted manner. The router learns its decisions along with everything else; nobody assigns fields of expertise by hand.
One should not picture the experts as school subjects. There is rarely one expert for chemistry and another for French. The division that emerges during training is usually barely interpretable for humans. What matters is only that it makes the model better overall.
Two problems accompany this design. First, the complete model must reside in the chips' memory, including the experts that are currently idle. One saves computing time, not memory — and this is exactly what makes such models difficult for home computers. Second, the router tends to overload favorite experts while others learn hardly anything. To counter this, training includes additional rules that enforce a more even distribution.
How to recognize Sparse MoE in product announcements
Announcements of new models often feature two numbers side by side. One states the total number of storage slots, the other the “active parameters.” If the second number is significantly smaller, it is a Sparse MoE model. Phrasings like “8 experts, 2 of which are active” mean the same thing.
For you as a user, the technology remains invisible. You only notice it indirectly: answers arrive faster, and powerful models are also available through free access tiers. Conversely, Sparse MoE explains why some powerful models nevertheless require enormous server hardware.
For clarity: the counterpart is called “dense.” There, the entire model computes for every single request. Dense models are simpler to build and run more stably, but scale more expensively. In technical discussions, the two approaches are therefore regularly weighed against each other.