Latent Mixture-of-Experts

Large language programs consist of millions of small computational building blocks. In the classic design, all building blocks are engaged for every single word. This is expensive, because each building block costs computation time and electricity. An alternative splits the model into groups, so-called experts, and activates only two or three of them per word. Latent Mixture-of-Experts is a variant of this idea: the decision as to which experts get to act is not made based on the raw word, but based on a compressed intermediate representation of that word. This compressed form is called latent, meaning hidden — it is a short list of numbers in which the model has condensed the meaning of the word.

What the detour via the intermediate representation achieves

The obvious benefit is money. A model can have hundreds of billions of parameters but use only a small fraction of them per request. Parameters are the adjustable numbers that a model learns during training. Computational costs arise only for the active parameters. This way, one gets the knowledge of a huge model at the price of a small one.

The second benefit is order. When the selection depends directly on the word, it often orients itself toward superficial features — the language, the spelling, common word forms. Two sentences with the same meaning then end up with different experts. The compressed intermediate representation largely ignores such surface features. This increases the chance that an expert actually covers a topic rather than merely a linguistic pattern.

An important distinction: Latent Mixture-of-Experts does not automatically make a model smarter. It makes a very large model affordable in ongoing operation. That is precisely why the term appears in quarterly figures and analyst reports when it comes to the operating costs of AI services.

Router, experts, and the problem of balance

The process has three stages. First, the model converts the incoming word into a long list of numbers that describes its meaning in the context of the sentence. This list is compressed down to a few dimensions — this is the latent representation. Then a small auxiliary network, the router, examines this short list and selects the appropriate experts.

No one manually determines what an expert is responsible for. The router learns its selection during training, together with all other parts. Only afterward can one sometimes recognize that an expert responds to, say, program code or numbers. Often, however, the specializations remain fuzzy and are hard to put into words.

The typical failure case is called load imbalance. The router sends almost everything to the same two or three experts, while the rest barely gets anything to do. The model then wastes memory space on unused building blocks. Developers therefore build an additional rule into the training that penalizes overly one-sided distributions. And one drawback remains in any case: all experts must reside in the graphics card's memory, even the ones currently idle. One saves computation time, not space.

Where this design is found in current models

Practically all large language models of recent generations rely on some form of expert partitioning. The approach became well known through Chinese providers, whose models were trained with comparatively small budgets. The models behind widely used chat programs likely work this way too, although manufacturers rarely disclose the details.

As a user, you don’t notice this directly. You see it indirectly in the prices: a model with an enormous scope of knowledge often costs only fractions of a cent per request. Tech news regularly attributes exactly this price development to Mixture-of-Experts architectures.

In technical articles, the term “latent” is also encountered in other contexts, for instance in latent spaces in image generation. It always refers to the same principle: a compressed numerical representation that is not directly readable but carries the meaning of the data.