PR-AUC

PR-AUC is a grade for programs that are supposed to fish something rare out of a huge pool. An example: a bank checks millions of transfers every day, and only a tiny fraction of them are fraud. A screening program outputs a suspicion score for every transfer. When evaluating it, two things matter: how many of the alarms are actual fraud? And how much of the total fraud did the program even notice? PR-AUC expresses both questions together in a single number between 0 and 1, where higher is better.

Why accuracy lies when cases are rare

The most obvious way to evaluate is accuracy: what percentage of all decisions were correct? For rare events, this number is worthless. If only one in a thousand transfers is fraud, a program achieves 99.9 percent accuracy by always saying “no fraud.” It would then have found not a single case of fraud and would still be nearly flawless. PR-AUC does not fall for this trick, because it does not count the many boring normal cases.

That is precisely why experts reach for PR-AUC when one class is strongly in the minority. Typical fields are fraud detection, medical screening, spam filters, and the search for defective parts in production. In all these cases, the cases being sought are rare but costly if overlooked.

A related metric is called ROC-AUC. It additionally takes into account how well the normal cases are recognized as normal. For extremely rare events, ROC-AUC often looks impressively high even though the program is practically useless. In such situations, PR-AUC is the more honest yardstick.

From two percentages to a curve

Behind PR-AUC lie two basic values. Precision asks: when the program raises an alarm, how often is it right? Recall asks: of all the cases that actually exist, how many did it flag? Both values depend on a threshold. If the program raises an alarm as soon as the suspicion score reaches 0.3, it finds a lot but produces many false alarms. If it waits until 0.9, the alarms are reliable, but it misses cases.

So one does not have to settle on a single threshold to evaluate the program. Instead, all possible thresholds are tried out. For each one, the pair of precision and recall is noted and plotted on a chart. The points together form a curve, the precision-recall curve.

PR-AUC is the area under this curve. If the curve bulges far toward the upper right, precision stays good even at high recall, and the area becomes large. An important difference from other metrics: the worst meaningful value is not 0.5, but the proportion of the rare class. With a fraud rate of one percent, a random program guesses a PR-AUC of 0.01. A value of 0.3 there would already be a very good result, even though the number looks small.

Where the number shows up in reports and products

In research papers on medicine and security, PR-AUC almost always appears in the results tables. Companies also cite it when presenting a new detection system, for example for credit card fraud or for suspicious network access. Anyone training programs themselves will find the metric in common toolkits under names like average_precision_score.

A common mistake is to directly compare PR-AUC values from different studies. This does not work, because the value depends on the proportion of the rare class. The same algorithm automatically achieves a higher number on a dataset with five percent fraud cases than on one with half a percent. A comparison only makes sense between programs that were tested on the same data.