Floating-Point Precision

Computers do not store numbers with unlimited precision. For decimal numbers like 3.14159, there are fixed formats that provide for a certain number of digits. Such a format is called a floating-point format, because the decimal point shifts depending on the size of the number. Floating-point precision describes how many digits a format provides. The more digits, the more exact the number, but the more storage space it requires. Shorter formats are less precise, but faster and more economical.

Why a few decimal places decide over millions of euros

A modern AI model consists of billions of individual numbers. These numbers are called parameters and store everything the model has learned. Each one of them has to be stored somewhere. For a model with 70 billion parameters, it makes a huge difference whether a number occupies four bytes or just one.

In numbers: at four bytes per parameter, the model needs around 280 gigabytes of memory. At one byte, it's only about 70 gigabytes. The first case requires several expensive specialized chips, the second fits on a single one. This means the price of hardware and electricity is directly tied to precision.

That's why precision has become one of the industry's most important levers. Companies like Nvidia explicitly advertise their chips based on how fast they compute in economical formats. Halving the precision roughly doubles the number of computational steps possible per second. The goal is always the same trade-off: as coarse as possible without the answers getting worse.

From 32 bits down to 8 bits

A floating-point number consists of two parts. One part determines the digits, the other the order of magnitude, i.e. how far the decimal point is shifted. A longer format allocates more space to both parts. This is exactly where the common formats differ.

For a long time, FP32 was the standard, i.e. 32 bits or four bytes per number. Today, training is mostly done with 16 bits, often in the BF16 format. For ongoing operation, one goes even lower, to 8 bits or even 4 bits. This subsequent shortening is called quantization.

You can imagine it like a map. A map at a scale of 1:10,000 shows every house, one at a scale of 1:1,000,000 shows only cities. For a trip from Hamburg to Munich, the coarse map is entirely sufficient. Similarly, an AI model often only needs a few digits, because it merely estimates probabilities. But if the format becomes too coarse, small differences disappear entirely, and the answers become noticeably flawed.

Where FP16 and FP8 show up in reports

Abbreviations like FP16, BF16, or FP8 appear in almost every press release about new AI chips. Performance figures always apply only to a specific format. A number without a format specification is therefore worthless, because the value in FP8 can easily be four times higher than in FP32. When comparing two chips, one must therefore check which format is meant.

The topic also comes up when downloading free language models. There, each file is labeled with an abbreviation like Q4 or Q8 for precision. The economical variants run on a normal gaming PC, the precise ones only on servers. Anyone who tries it themselves notices the difference: heavily shortened models often answer less accurately or lose the thread.

A common misconception is equating precision with quality. A well-quantized large model is usually better than a small model at full precision. What matters is not the number of digits, but what ultimately comes out in the answers.