Sparse Mixture of Experts
Sparse Mixture of Experts is a construction principle for large AI models: the model consists of many interchangeable subnetworks, of which only a few compute per request. This allows a model to be very large while still responding quickly and cheaply.
Large language programs like ChatGPT consist of millions of small computational building blocks. Normally, every single one of these blocks is involved in the computation for each question. This costs a lot of electricity and time. Sparse Mixture of Experts takes a different approach. The program is divided into many similar subnetworks called experts. For each request, only one or two of these experts are switched on, while the rest remain idle. “Sparse” describes exactly that: only a small portion computes.
Why large models would be too expensive without this principle
The capabilities of an AI model depend heavily on its size. Size is measured in parameters, i.e., in adjustable numbers inside the model. More parameters generally mean better answers. But they also mean more computational work per question. This is precisely where operators hit an economic limit.
Sparse Mixture of Experts breaks this coupling. A model can have one trillion parameters, of which only thirty billion are used per request. In operation, it behaves like a small model, but in terms of knowledge like a large one. Well-known models such as Mixtral, DeepSeek-V3, or presumably GPT-4 use this principle. Without it, chatbots for millions of users would hardly be affordable.
A side effect concerns training. Even during learning, only a fraction of the model needs to work. Companies therefore get a significantly larger model for the same budget. This explains why the method has appeared in almost every new top-tier model since around 2023.
The router decides who gets to compute
The selection of experts is handled by a small additional network, the router. It is presented with each piece of text individually and assigns scores to all experts. The two highest-scoring experts are allowed to compute, and their results are mixed together. The others sit out this round. This happens anew for each word, not just once per question.
No one determines in advance what an expert is responsible for. The router learns its decisions together with the rest of the model. In the end, experts sometimes specialize in program code, sometimes in numbers or punctuation. Often, however, it’s impossible to say what pattern an expert follows. The term “expert” is therefore more of a metaphor than a precise description.
In practice, the balance is tricky. Without countermeasures, the router sends almost everything to the same two experts, because these happened to be slightly better at the start by chance. The remaining experts then never learn anything. Developers therefore add a penalty rule into training that punishes uneven distribution. A second catch remains: the entire model must reside in the memory of the graphics cards, including the idle experts. What is saved is computing time, not space.
Sparse MoE in model names and figures
The term rarely appears in marketing but frequently in technical reports on new models. A typical telltale sign is two parameter figures side by side. In Mixtral 8x7B, the numbers stand for eight experts with seven billion parameters each. When a report speaks of “active parameters,” it is almost always Sparse Mixture of Experts behind it.
Even as a user, one notices the effects without hearing the name. Answers arrive faster even though the models are computationally ever larger. And prices per request have fallen sharply for many providers within two years. A good part of this price drop is due to this construction principle.
A common misconception is that Sparse Mixture of Experts is a group of separate programs that consult with each other. This is not the case. It is a single model whose inner workings are merely divided into sections. Those who have multiple independent AI systems work together instead speak of agents or of an ensemble.