Precision

A computer program is supposed to find all the advertising emails among a thousand emails. It flags a hundred emails as advertising. Eighty of them are indeed advertising, twenty are harmless messages from friends. In this case, precision is 80 percent, that is, 80 out of 100 reports. Precision thus answers exactly one question: when the program raises an alarm, how often is it right? It says nothing about how many advertising emails the program missed.

What a false alarm costs

A single percentage figure rarely suffices to describe the quality of a system. One can very easily configure a program so that it flags almost everything. Then it does find every advertising email, but it also sorts out many important messages. Precision makes exactly this kind of error visible.

How bad a false alarm is depends on the use case. With a spam filter, a job application might vanish into the trash folder. With medical image recognition, a false alarm means an unnecessary operation. With police facial recognition, it means an innocent person gets stopped and checked. In such areas, high precision is demanded, even if the system thereby misses more real cases.

It is also important to note a common misconception: high precision is no proof of a good system. A program that reports only a single, completely obvious case and gets it right has 100 percent precision. It is nevertheless useless, because it misses almost everything.

The calculation with the four boxes

To calculate it, all decisions are sorted into four groups. True positive means: reported and actually correct. False positive means: reported, but in reality harmless. In addition there are the missed cases and the correctly ignored cases. Precision is the number of true positive reports divided by all reports, that is, true positive plus false positive.

The counterpart to precision is called recall, or hit rate. It asks: what proportion of all actually existing cases did the system find? Between the two figures there is almost always a trade-off. If a system becomes more cautious, precision rises and recall falls. If it becomes bolder, it is the other way around.

This can be controlled via a threshold. Many models do not output a fixed answer but a probability, for instance “87 percent spam.” It is only the operator who determines the value above which something is reported. A high threshold of 95 percent yields a lot of precision, a low one of 50 percent yields many hits. Because a single number is often misleading, experts usually state both values or combine them into the so-called F1 score, an average of precision and recall.

Precision in product announcements and studies

Anyone reading reports about new AI systems constantly comes across such figures. Manufacturers of fraud detection systems for banks advertise precision values, because every false suspicion means an angry customer. In scientific publications on image recognition or search engines, precision and recall almost always appear side by side in a table.

The effect is also noticeable in everyday life. The spam folder in the inbox, the suggestions of a search engine, the content filters of a video platform: everywhere a threshold has been set, and thereby a decision about precision has been made. If a platform blocks harmless videos, its filter is set too imprecisely.

When reading such figures, it is worth asking a follow-up question. A precision of 99 percent sounds impressive, but says little as long as the recall is missing. And it always applies only to the test data with which it was measured. In practice, with different users and new fraud schemes, the values often turn out considerably worse.