MoE Architecture
The MoE architecture is a blueprint for large AI models in which not the entire model computes, but only a few specialized subsections per request. This allows a very large model to respond as quickly as a much smaller one.
An AI program like a chatbot consists internally of huge tables of numbers. When you ask a question, your input is run through these tables, and in the end an answer emerges. In many models, every single number is involved in this computation. That costs a lot of computing time and thus a lot of money. The MoE architecture does it differently: it splits the tables into many subgroups called experts. For each request, only a few of these groups are used, while the rest remain idle.
Why large models without MoE would become too expensive
A model’s capabilities depend heavily on how many adjustable numbers it possesses. These numbers are called parameters, and more parameters usually mean better answers. This is exactly where the problem lies: if all parameters have to participate in the computation for every request, costs rise at the same pace as the model’s size. A model with ten times as many parameters would then also be roughly ten times as expensive to operate.
The MoE architecture breaks this coupling apart. A model can have 500 billion parameters in total, but activate only 30 billion of them per word. Experts therefore distinguish between total parameters and active parameters. For response speed, only the active ones count; for the stored knowledge, it’s more the total number that matters.
This is the reason why many of the strongest models of recent years are built on this principle. Providers can thereby operate very large models without every single answer becoming unaffordable. Operation on fewer graphics cards also becomes more realistic, because fewer computational steps are needed.
The router and the distribution to experts
The central role is played by a small additional network, the so-called router. It looks at each token and decides which experts are responsible for it. Typically, two out of perhaps 64 or 128 experts are selected. Their results are then combined and passed on.
Importantly: no one defines the areas of expertise by hand. The router is learned along with the rest of the model during training. What an expert ends up being good at emerges on its own from the training data. Often the resulting specializations do not correspond to human categories like mathematics or French, but are barely comprehensible to us.
A common misconception is that MoE also saves storage space. That’s not true. All experts must remain loaded, because a different one might be needed for the next word. What is saved is computing time, not memory. Another problem is load balancing: if the router favors certain experts, some become overloaded while others learn almost nothing. That’s why additional rules are built into training that reward even utilization.
MoE in current models and news
The term appears mainly in technical descriptions of new language models. When a company writes that its model has '671 billion parameters, of which 37 billion are active,' then it’s an MoE model. The principle became known to a wider public through models from the French provider Mistral and through the Chinese DeepSeek models, which achieved very low operating costs with it.
As a user, you don’t notice the architecture directly. It has an indirect effect: on the price per request, on response speed, and on which models can be offered for free at all. The topic is also relevant on the stock market, because MoE models change the demand for expensive computing power.