MXFP8

At its core, an AI model consists of billions of numbers. A computer has to store these numbers, and that requires space. How much space a single number takes up is determined by the so-called number format. MXFP8 is a particularly economical format: in it, each number occupies only eight bits, meaning eight tiny memory cells, each holding a zero or a one. By comparison, the classic format used for such calculations needs 32 bits per number, four times as much. The “MX” in the name stands for Microscaling and refers to a trick that keeps the format surprisingly accurate despite the few bits.

Why eight bits instead of 32 makes the difference

Memory on modern AI chips is scarce and expensive. A large language model with hundreds of billions of numbers doesn’t fit on a single graphics card in 32-bit format. Halving the bits per number also halves the memory requirement. With MXFP8, it shrinks to a quarter. Models that previously needed twenty chips then run on five.

The second benefit is speed. Chips like Nvidia's Blackwell generation have built-in dedicated computing units for eight-bit numbers. These units perform roughly twice as many computing operations per second as those for 16-bit numbers. For companies, this translates directly into money: a training run that would otherwise take three months might be finished in six weeks.

What makes MXFP8 particularly interesting is that it doesn’t just work when using the finished model. Earlier eight-bit formats were usually only suitable for ongoing operation. During training itself they were too imprecise, and the model learned worse or not at all. MXFP8 is accurate enough to carry out training with it as well.

The shared scaling factor per block

With eight bits, only 256 different values can be represented. That’s not many. But the numbers in an AI model range from tiny to fairly large. If you squeeze them all into the same 256 steps, the small values get rounded to zero and are lost. This is exactly where microscaling comes in.

MXFP8 divides the numbers into blocks of 32 each. Each block gets its own shared scaling factor, which is stored additionally. The eight bits of a number therefore don’t describe the value directly, but rather its ratio to this block factor. You can think of it like a scale on a map: the same centimeters sometimes mean kilometers, sometimes meters, depending on what’s shown alongside them.

The advantage: a block full of small numbers gets a small scale and retains its fine distinctions. A block with large numbers gets a large one. The extra overhead is minimal, since one factor is enough for 32 numbers. On average, this costs less than one additional bit per number in computational terms. A related format called MXFP4 uses only four bits per number. It’s even more economical, but noticeably less accurate and so far barely suitable for training.

MXFP8 in chip announcements and model reports

In everyday life, you never see MXFP8 directly. It’s embedded beneath the surface of every chatbot and every image generation running on modern hardware. It only becomes noticeable indirectly, for instance through shorter waiting times or lower prices for using AI services.

In the news, however, the term comes up regularly. Chip manufacturers like to cite performance figures for new processors such as “20 petaflops in FP8”. Such figures refer to eight-bit formats like MXFP8 and therefore appear more impressive than figures given in 16 or 32 bit. So when comparing two chips, it’s worth looking at which format is meant.

A common misconception is that MXFP8 fundamentally makes models dumber. With careful implementation, the quality loss is in the range of a few tenths of a percent and is barely measurable in tests. The format is also not a solo effort by one company: it was established as an open standard by a consortium including Nvidia, AMD, Intel, Microsoft, Meta, and others. That’s why it can now be found in hardware from a wide variety of manufacturers.