Conformal Prediction

Computer programs that learn from examples usually give exactly one answer. An image recognition program might say: “That’s a cat.” How confident it actually is often remains unclear. Conformal Prediction changes this output. Instead of a single answer, the method delivers a selection of possible answers, for example “cat or lynx.” This selection is calculated so that it contains the correct answer in a predetermined proportion of cases — for example, in 95 percent.

A guarantee that holds even with poor models

Many learning programs output percentages, such as “87 percent cat.” These numbers sound like probabilities, but they are often too optimistic. Experts call this poorly calibrated values: the program claims 99 percent certainty and still regularly gets it wrong. Such numbers cannot serve as a basis for decisions.

Conformal Prediction sidesteps this problem because it measures reliability after the fact instead of taking it on faith. The method checks against real data how often the model was wrong in the past. From this, it derives how wide the answer set needs to be. The promised hit rate then holds even if the underlying model is mediocre. A weak model does not become smarter as a result — its answer sets simply become larger, and thus more honest.

This is especially valuable wherever mistakes are costly. In medicine, credit decisions, or industry, one wants to know when a machine cannot commit to a single answer. An answer set with five possibilities is a clear signal: a human should take a closer look here.

The calculation path with the calibration stack

The method requires its own dataset, the so-called calibration stack. These are examples with a known correct answer that the model never saw during training. For each of these examples, one calculates how poorly the model performed. This value is called the nonconformity score and is often simply the distance from the truth.

These values produce a ranking of errors. If one wants a hit rate of 95 percent, one looks for the value below which 95 percent of all errors fall. That is the threshold. For a new query, one then includes in the set all answers whose error value stays below this threshold.

You can think of it like a clothing size that isn’t estimated but measured against a hundred customers. The rule stays the same, only the fit is based on real measurements. One condition is important: the new data must resemble the calibration data. If reality shifts significantly, for example because a camera was replaced, the guarantee loses its validity.

From the weather forecast to the chatbot

The basic idea is older than the buzzword. When the weather service names a temperature range of 18 to 22 degrees, it is working with the same idea: a range instead of a number. Conformal Prediction turns this into a method with a solid mathematical foundation that can be applied on top of any learning program.

In products, one usually encounters the term in the background. Medical image analysis uses it to flag suspected cases for referral to physicians. In quality control, it decides which parts a human needs to inspect. Research is also underway for language models: the system should recognize when its answer stands on shaky ground.

In news coverage, the term often appears together with “uncertainty quantification,” that is, the question of how confident a machine is in its own answer. A common misconception: Conformal Prediction does not make predictions more accurate. It only makes visible how inaccurate they are — and that is precisely its practical value.