Sparse FP8

An AI model contains billions of numbers, known as parameters. The more decimal places stored for each number, the more precise — but also the more expensive — the computation. FP8 is a format that shrinks numbers down to eight bits — that is, eight ones or zeros. For comparison: older standard formats use 16 or 32 bits per number. In addition, the “sparse” property comes into play: with sparsity, values close to zero are simply treated as zero and skipped during computation. Sparse FP8 combines both — coarse numbers plus targeted skipping — in a single method.

What Sparse FP8 means for AI costs

AI models aren’t trained just once — afterward, they respond to queries millions of times a day. Every response costs computing time, and therefore electricity. Anyone operating a chatbot pays for every single output. For large providers, this adds up to enormous sums.

Sparse FP8 intervenes in two places at once. Smaller numbers mean less data transfer between memory and the compute unit — which is often the actual bottleneck. And skipping zeros means fewer computing operations. Nvidia states that for its Blackwell chips, Sparse FP8 can enable up to four times the computing performance compared to conventional formats. Even if everyday performance falls somewhat short of that, it’s still a substantial lever.

A common misconception is that cheaper computation automatically delivers worse results. That’s not necessarily true. Many values in a model are already so small that they barely contribute to the answer. Omitting them or rounding them coarsely barely worsens the output — but saves considerable resources.

How the combination of shrinking and skipping works

FP8 stands for “8-bit floating point,” meaning a floating-point number with eight bits. Normal floating-point numbers in the FP32 format have 32 bits and can represent very fine differences. FP8 forgoes this level of detail. This is a deliberate trade-off: a small loss of precision is accepted in order to save memory and computing time.

The sparsity side works like this: in a typical neural network — a network made up of many layers that learns from examples — many connections are weak after training. Their weight, meaning their influence on the result, is close to zero. The hardware can be built to recognize such zeros and skip the corresponding computation step. At 50 percent sparsity — the common target value — every second value is skipped, which halves the computing operations.

For this to work, three things must align: the model must actually contain enough zeros, the data format must support FP8, and the hardware must be able to handle structured sparsity. Nvidia’s current graphics processors from the Hopper and Blackwell generations are designed precisely for this. Without matching hardware, the format offers no advantage.

Sparse FP8 in current models and chips

Sparse FP8 shows up mainly in reports about new chip generations and large language models. When Nvidia introduces a new graphics processor, the company almost always cites a peak compute figure in FP8 — this figure already assumes sparsity. Without that caveat, the number would be significantly smaller. So it’s worth taking a close look, when reading chip benchmarks, at which format and which sparsity level are being specified.

FP8 is also increasingly being used in the training of large models, not just in later operation. Meta and Google have publicly described carrying out parts of their models' training in FP8 to cut costs. This was long considered unthinkable, because too much precision seemed likely to be lost — but researchers have since developed techniques that keep the loss manageable.

In everyday life, Sparse FP8 isn’t encountered directly — you don’t see it when using a chatbot. It works in the background: as one of the reasons why AI responses arrive faster and providers don’t have to make their services even more expensive than they already are.