FP8 Precision

At its core, an AI model consists of vast quantities of numbers. These numbers must be stored in a computer’s memory, and there are various formats of differing frugality for doing so. FP8 is a particularly frugal format: each individual number occupies only eight bits, that is, eight zeros or ones. By comparison: for a long time it was standard to store each number with thirty-two bits, later with sixteen. The letters FP stand for “floating point” — a format that can represent both very small and very large numbers. The price for this frugality is accuracy: eight bits accommodate only about 256 different values.

Why eight bits determine cost and speed

In practice, large AI models almost never fail due to computing power alone. They fail because the numbers don’t fit into the memory of the graphics cards. A model with 70 billion numbers occupies around 140 gigabytes in 16-bit format. In FP8, it’s only about 70 gigabytes. This suddenly allows it to fit on a single machine instead of two.

The second gain is speed. The path from memory to the computing unit is often the bottleneck in AI computations. Numbers half the size mean that twice as many of them can be transported in the same amount of time. Modern chips also feature circuits specifically built for 8-bit numbers, which process them significantly faster than 16-bit numbers.

For companies, this translates directly into money. Anyone operating a chatbot pays for electricity and computing time for every single response. Several major models in recent years have already been trained entirely in FP8, including models from DeepSeek. The operators reported significant savings in training costs as a result.

How to make do with 256 values

A floating-point number consists of two parts. One part determines the order of magnitude — that is, whether we’re talking about thousandths or millions. The other part determines the exact decimal digits. With FP8, there are two common ways of splitting up these eight bits. One variant sacrifices decimal precision in favor of magnitude range, the other does the opposite.

You can picture this like a ruler with room for only 256 marks. If all the measured values lie close together, the scale is fine enough. If a single outlier lies far outside the range, the ruler must be stretched, and all other values lose resolution. This exact problem occurs constantly in AI models.

The solution is called scaling. Before the computation, the numbers in a group are multiplied by a shared factor so that they fill the available value range well. After the computation, the factor is divided back out. In addition, sensitive parts of the model are deliberately left running in 16 bits. A common misconception is that FP8 replaces all other formats — in practice, it is almost always a mixture.

FP8 in chip announcements and model data sheets

The term most commonly appears in news about graphics cards and AI chips. When Nvidia or AMD unveil a new generation, they typically list performance figures separately for different formats. The most spectacular figures are almost always listed alongside FP8 or even smaller formats. Anyone comparing such numbers should check whether the same format is actually being referenced.

FP8 also appears in the technical descriptions of freely available models. Providers often offer their models in multiple versions, for instance in 16-bit and in FP8. The smaller version runs on cheaper hardware but delivers slightly different results.

FP8 should be distinguished from formats like INT8 or INT4. These work with whole numbers without a magnitude component and are usually applied only after the fact to a finished model. FP8, by contrast, is precise enough to be used during training itself. This is the decisive difference that has made the format so important since around 2023.