calibrated probability

Many AI models don’t just output an answer, but also a percentage value expressing how certain the model is. A model is calibrated when this percentage value corresponds to reality. If a model says “I am 70% certain” across a hundred different questions, then it should actually be correct in approximately seventy of them. If that’s the case, the output percentage values are called calibrated probabilities. If it’s not the case, the model is poorly calibrated — and its confidence statements are misleading, even if the answers themselves are often correct.

Why poor calibration is dangerous

A model can be correct very often and still be poorly calibrated. That sounds contradictory, but it isn’t. Imagine a model that always reports “99% certain” for medical diagnoses — even when it’s only correct 60% of the time. A doctor who trusts these values makes worse decisions than if they had received no percentage figure at all.

This problem occurs especially with so-called neural networks, the computational systems behind modern AI models. They tend to be systematically overconfident: they say “95% certain” more often than their actual accuracy rate would justify. Conversely, there are also underconfident models that trust themselves too little. Both distort the decisions of the people who deploy the model.

How calibration is measured and improved

To check whether a model is calibrated, one collects many predictions and groups them by their stated confidence. All predictions with about 80% confidence go into one group, all with about 60% into another, and so on. Then one looks at how high the actual accuracy rate is in each group. The result can be displayed in a so-called reliability diagram: a graph in which a perfectly calibrated curve appears as a straight diagonal. If the curve deviates significantly, the model is poorly calibrated.

There are mathematical post-processing methods for improvement. The best known is called Platt scaling: one takes the model’s raw output values and converts them into better-fitting percentage values using a simple formula. The model itself is not changed in the process — essentially, a correction layer is attached afterward. Another method is called temperature scaling and works similarly, but with a single adjustable numerical value that compresses or stretches the model’s outputs until they match reality.

Important: these methods do not improve the model’s accuracy. The answers do not become correct more often. Only the confidence statements become more honest.

Where calibrated probabilities show up in practice

In medicine, calibration is a legally relevant topic. AI systems that support doctors in diagnoses must meet certain quality requirements in the EU under the AI Act. A systematically overconfident AI that outputs “98% probability of tumor” or “98% probability of no tumor” for every X-ray image would pose a significant risk.

Calibration has also been standard in weather forecasting for decades. When a weather service reports “30% chance of rain,” it means that historically, it has rained in roughly 30 out of 100 cases under comparable weather conditions. This is an example of good calibration outside of AI.

In tech reporting, the term mainly comes up when researchers demonstrate that a well-known language model is poorly calibrated despite impressive answers. This is an important objection to the use of such models in areas where people must trust the stated figures — such as legal advice, financial analysis, or education.