FLOPs

Computers are constantly calculating with numbers that have a decimal point, for example 3.14 times 0.7. Every single such computational step is called a floating point operation, abbreviated FLOP. The plural of this is FLOPs. It is used as a unit of measurement, similar to kilometers for distances or kilowatt-hours for electricity. Two things are measured with it: how much computational work a task costs in total, and how many computational steps a chip can perform per second. The second meaning is often written FLOP/s, though unfortunately the two are constantly confused in everyday language.

The currency in which AI is paid

A modern AI model learns by repeatedly adjusting billions of numbers slightly. Each of these adjustments consists of a great many individual multiplications and additions. In the end, this adds up to astronomical values. For training large language models, magnitudes of 10 to the power of 25 FLOPs and more are mentioned today. That’s a one followed by 25 zeros.

This number is interesting because it translates directly into money and electricity. Computational steps require chips, chips require time and energy. Anyone who knows how many FLOPs a training run costs can roughly estimate what it consumes in data center rental costs. Experts therefore speak of a project’s compute budget, measured in FLOPs. It is the scarcest resource alongside good data.

Politics has also discovered the term. The EU AI Act and earlier requirements in the US define threshold values in FLOPs. If a training run exceeds a certain limit, stricter reporting and review obligations apply. The idea behind this: beyond a certain amount of compute, a model could become powerful enough that regulatory scrutiny becomes appropriate. However, whether a pure compute figure is the right measure for this is fiercely debated.

How the number is actually determined

There is a well-known rule of thumb for the effort of a training run. You roughly calculate six times the number of parameters times the number of training words. Parameters are the adjustable numbers inside the model that store what has been learned. A model with 10 billion parameters, trained on 2 trillion words, thus comes out at about 10 to the power of 23 FLOPs. The formula is rough, but entirely sufficient for comparisons.

For chips, on the other hand, speed is what counts. A graphics processor from a data center today can perform several hundred trillion computational steps per second. This figure appears on every data sheet and is gladly cited in advertising. However, it is a theoretical maximum under ideal conditions. In practice, a training run often achieves only thirty to fifty percent of that, because the chips have to wait for data from memory.

A common misconception is that more FLOPs automatically means a better model. Compute power is an ingredient, not the recipe. Poor data or a clumsy model architecture will fizzle out even with a huge budget. Conversely, clever methods in recent years have ensured that equally good results can be achieved with significantly fewer computational steps.

FLOPs in data sheets and headlines

You most often encounter the term in reports about new chips. When Nvidia or AMD unveil a new generation, the FLOP/s figure sits right at the top of the press release. Rankings of the world’s fastest supercomputers are also kept in this unit. The frontrunners are now in the range of trillions upon trillions of computational steps per second—referred to as exascale.

In reports about AI companies, the unit appears indirectly. When there is talk of billions in investment for data centers, this ultimately concerns the purchase of FLOPs. Some companies publish technical reports on their models in which the training effort is disclosed. Others remain silent about it, because the number allows conclusions to be drawn about costs and strategy.

A practical tip for reading such reports: pay attention to whether a second is mentioned. FLOPs without a time reference represent a total amount of work, FLOP/s represent a speed. And pay attention to the precision of the numbers being used in the calculation. A chip can perform far more operations per second with more coarsely stored numbers than with very precise ones. Manufacturers therefore like to cite the highest figure that can somehow be justified.