Sparsity

At its core, an AI model consists of huge tables of numbers. These numbers are multiplied and added together millions of times with every request. Sparsity describes the state in which a large portion of these numbers is zero. Multiplications by zero always yield zero, so they can be skipped. The opposite is called densely populated: there, almost every number differs from zero, and each one must actually be computed. However, the term also refers to a second thing, namely that only a small part of the model is switched on at all per request.

Why zeros save money

Operating large AI models costs one thing above all: computing time on specialized chips. These chips are expensive, scarce, and consume a lot of power. Every computing operation saved therefore directly lowers the cost per answer. For systems with millions of users, this adds up to amounts in the millions.

Then there’s memory. A table in which nine out of ten entries are zero doesn’t need to be stored in full. It’s enough to store the values that differ from zero along with their positions. It’s like a class schedule where you only note the occupied hours instead of every empty slot. This is how models fit onto devices that would otherwise be too small.

One reason for the enthusiasm is the observation that sparsity often does little harm. Large models contain a lot of redundancy. A considerable share of the connections can be removed without the answers noticeably getting worse. Sometimes the model even becomes more stable as a result, because it becomes less prone to memorizing training examples.

How models are made sparse

The simplest way is called pruning. First, a normal, densely populated model is trained. Afterward, the numbers with the smallest magnitudes are identified and set to zero. Then a brief round of retraining follows, so that the rest adapts to the gaps. The image of trimming a tree fits well: the thin branches fall away, the load-bearing boughs remain.

The second form doesn’t work with zeros but with switches. In a mixture-of-experts model, the network is broken down into many sub-networks called experts. A small additional network decides, for each word, which two or four experts are allowed to respond. All others stay silent. Such a model can have hundreds of billions of parameters and still compute as fast as a much smaller one.

There’s a catch that’s easy to overlook. Randomly distributed zeros do little good on real hardware, because chips compute in fixed blocks. That’s why modern graphics cards require structured patterns, for instance exactly two zeros in every group of four. And with the expert approach, the entire model must reside in memory, including the parts that are currently inactive. What’s saved is computing time, not space.

Sparsity in products and headlines

Reports about new AI models often list two parameter counts side by side. These are the total number and the number of active parameters per request. If the second number is far below the first, sparsity is behind it. This very principle explained part of the low training and operating costs with which Chinese providers caused a stir in 2025.

You also encounter this effect on your own phone. Voice assistants, translation, and camera features increasingly run directly on the device. This is made possible by pruned models that, together with more coarsely stored numbers, need only a fraction of the memory. Without such techniques, these requests would have to go to a data center.

For investors and observers, sparsity is therefore a keyword with economic significance. It helps determine whether a provider can operate its models profitably. However, a common misconception is that sparsity fundamentally makes models smaller. Above all, it makes them cheaper to operate, while the models as a whole keep growing.