Model FLOPs Utilization

A computer chip that trains AI models can perform a certain maximum number of computing operations per second. This maximum figure is stated in the manufacturer’s data sheet and represents a theoretical best value. In reality, no software reaches this value because the chip repeatedly has to wait for data. Model FLOPs Utilization is the metric that puts both figures in relation to each other. It tells you: this percentage of the possible computing power was actually used for calculating the model. The rest was lost to waiting time, data transport, or administrative tasks.

What a percentage point of utilization costs

Training large AI models is one of the most expensive undertakings in the tech industry. For a very large language model, tens of thousands of specialized chips run for weeks. The bill for this runs into the tens or hundreds of millions. The price depends almost directly on how long the chips are occupied.

This is exactly where the metric turns into money. If utilization rises from 30 to 45 percent, training finishes about a third faster. The hardware remains the same; not a single additional chip was purchased. Such improvements arise purely from better software and smarter distribution of the workload.

That’s why MFU is also a comparison figure between providers. If two companies use the same chips but achieve different utilization, one of them is working more efficiently. The number now appears regularly in technical reports and investor presentations. It is an indicator of how well a team masters its infrastructure.

Why chips wait so often

The calculation is basically simple. You count how many computing operations the model needs for one training step. This number is divided by the elapsed time, yielding the actual computing performance. The result is then divided by the chip’s peak value from the data sheet. The output is a percentage.

The loss almost always occurs at the same point: the supply of data. A computing chip can process numbers much faster than memory can deliver them. You can imagine it like a kitchen with an extremely fast cook, but only one delivery person on a small bicycle. The cook is not the problem; the supply is.

On top of that comes the communication between thousands of chips. Each of them computes only a part of the model and must synchronize its intermediate results with the others. During this exchange, no one is computing on the model. Engineers therefore try to overlap computation and data exchange in time. This is exactly where well-optimized and poorly optimized training runs differ.

MFU in technical reports and confusion with hardware utilization

The term rarely appears in marketing texts, but frequently in technical reports on new models. Major labs often publish a table with chip count, training duration, and achieved utilization when introducing a model. Cloud data center providers also cite the figure when they want to show that their systems operate efficiently.

A common misconception is confusing it with the chip utilization display in the operating system. This often shows 100 percent even though MFU is at 35 percent. The reason: the display only shows that the chip is busy. It does not distinguish between useful computational work and waiting for data.

It is also important that the metric primarily relates to training. During later operation of a model, when it answers individual user queries, the values are often significantly lower. There, the chip is waiting for data from memory almost continuously. For this case, experts use other metrics, such as responses per second or cost per request.