Sparse Architecture
A sparse architecture is an AI model that, for each individual request, uses only a small portion of its building blocks instead of all of them. This allows a very large model to compute as if it were much smaller.
An AI model consists of many billions of small computational values, called parameters. These are what the model has stored while learning from example texts. In a classic model, all of these values are computed for every single request. A sparse architecture does it differently: it switches on only a small section of the model per request, while the rest remains idle. “Sparse” means that in a single computational pass, most positions in the model are set to zero. The counterpart is called “dense”: there, everything is always active.
Why large models would otherwise be unaffordable
In dense models, size and cost are tightly linked. Doubling the number of parameters roughly doubles the computational effort per response. Larger models are usually smarter, but they are correspondingly more expensive to run. It is precisely this coupling that sparse architecture cuts through.
It separates two things that previously seemed inseparable: stored knowledge and effort per response. A model can contain a huge amount of knowledge without every single question having to sift through all of it. Some current models have several hundred billion parameters but activate only a few billion of them per word. The computational effort is then a fraction of what the total size would suggest.
For companies, this is the difference between a research project and a product. A chatbot with millions of users costs electricity and computing time anew with every response. The fact that modern AI services are often usable for free or for a few euros a month is directly tied to such savings tricks.
The router decides who gets to compute
The best-known design is called Mixture of Experts, or MoE for short. The model is broken down into many parallel sub-networks called experts. A small additional network, the router, selects one or two suitable experts for each word. Only these compute; all others remain idle.
Importantly, no one manually determines what an expert is responsible for. The router is trained together with the rest of the model and develops its own division of labor. What patterns an individual expert ultimately covers often cannot be cleanly named. There are other forms of sparsity as well, such as the subsequent removal of unimportant connections, known as pruning.
The approach has a catch worth knowing. All experts still have to reside in GPU memory at the same time, even the ones currently idle. What is saved is computing time, not memory space. And if the router distributes things poorly, individual experts end up constantly overloaded while others barely learn anything.
How to spot sparse models in the news
In model announcements, two numbers now almost always appear. First the total number of parameters, then the number of “active” parameters. If it says something like “400 billion total, 17 billion active,” it’s a sparse architecture. The second number says more about speed and cost than the first.
Major model families from Mistral, Google, DeepSeek, and Meta rely on this principle. It is also widely assumed to be the case for OpenAI’s top models, though this has not been officially confirmed. A common misconception in discussions: sparse doesn’t mean the model can do less. It just computes more selectively.
This shows up indirectly in everyday use. Responses arrive faster, and powerful models run on less hardware. Anyone trying out a smaller AI locally on their own computer benefits from this too. Sparse architectures are one of the reasons usable AI is no longer possible only in huge data centers.