Active Parameters in Mixture-of-Experts

An AI model consists of many billions of numbers that were tuned during training. These numbers are called parameters, and they are the model’s actual knowledge. In classical models, every response involves computation with all of these numbers. However, there is a design in which the model is broken down into many sub-networks called experts. For each request, only a few of these experts are switched on, while the rest remain idle. The numbers that actually participate in the computation are called active parameters.

Why two numbers appear in the spec sheet

For such models, announcements always list two figures. The first is the total number of all parameters, say 400 billion. The second is the number of active parameters, say 20 billion. Both numbers describe the same model, but they answer different questions.

The total figure indicates how much knowledge is stored in the model and how much memory it requires. The active figure indicates how much computational work a single response costs. For operating costs, it is almost exclusively the second figure that matters. A model with 400 billion parameters and 20 billion active ones computes roughly as fast as a small model with 20 billion parameters.

This is precisely the appeal of this design. You get the capabilities of a very large model at the computational price of a medium-sized one. That’s why providers like to advertise the low active parameter count. A common misconception is to conclude from this that the model will also run on a laptop. That’s wrong, because memory requirements are determined by the total count.

How the selection of experts comes about

Selection is handled by a small additional network called the router. It evaluates every word passing through the model and sends it to the appropriate experts. Typically two to eight active experts are chosen from a pool of dozens or hundreds. The router learns its assignment on its own during training; nobody divides subject areas by hand.

However, active parameters include not only the selected experts. Some components are always used, such as the layers that convert words into numbers, and the attention layers that establish word relationships within a sentence. These fixed portions are added to the selected experts. That’s why the active count is never simply the total divided by the number of experts.

A comparison helps: a large library has a hundred thousand books; for a homework assignment, you take three off the shelf. The three books are the active parameters, the building is the total count. The building still has to be heated, even if you only read three books. Translated to models: the graphics memory has to hold the entire model, even though only a fraction of it is working.

Where the figure appears in model announcements

Practically every new model family from major providers now uses this design. In announcements you read phrasing like “37 billion active out of 671 billion parameters.” Such figures come, for example, from the Mistral, DeepSeek, or Qwen models. Even for the large commercial systems, experts assume this construction, even though manufacturers don’t always publish the figures.

For investors and industry observers, the figure is a cost indicator. Falling active parameter counts at equal performance mean cheaper responses. This explains why prices per request have dropped sharply in recent years. Anyone reading news about new models should therefore always check which of the two figures is being cited.

Active parameters should not be confused with the context window, i.e., the amount of text a model can take in at once. One describes the model’s design, the other its input. Both figures often appear in the same paragraph of a press release. But they have nothing to do with each other.