F1 Score

Programs that learn from data often make simple decisions: Is this email spam or not? Does this X-ray show a fracture or not? Two quite different errors can occur here. The program raises an alarm even though nothing is wrong. Or it misses a case where it should have raised an alarm. The F1 score compresses both types of errors into a single number between 0 and 1. The closer to 1, the better the program performs.

Why accuracy alone can be misleading

The most obvious metric would be: What percentage of all decisions were correct? This figure is called accuracy. It sounds reasonable, but it can be grossly misleading. This happens whenever the case being sought is rare.

An example: Out of 10,000 people, 10 have a certain rare disease. A program that stubbornly outputs “healthy” for every person is correct in 9,990 out of 10,000 cases. That’s 99.9 percent accuracy. Yet the program is completely useless, because it doesn’t find a single sick person. The F1 score of this program would be 0.

This is exactly why experts turn to the F1 score whenever hits are rare. Credit card fraud detection, spam filters, searching for tumors in images: in all of these areas, the class of interest is a tiny minority. So if a study or press release only mentions accuracy, it’s worth asking a skeptical follow-up question.

Precision, recall, and their harmonic mean

The F1 score is composed of two sub-metrics. The first is called precision: Of all the cases where the program raised an alarm, how many were genuine? It thus penalizes false alarms. The second is called recall: Of all the genuine cases, how many did the program find? It penalizes missed cases.

Both values can easily be gamed if considered individually. Anyone who only raises an alarm for absolutely certain cases gets a high precision but misses a lot. Anyone who raises an alarm at every suspicion finds everything, but generates mountains of false alarms. So there is a trade-off between the two figures.

The F1 score combines them via what is known as the harmonic mean. The formula is: 2 times precision times recall, divided by the sum of precision and recall. What’s special about this: the harmonic mean pulls strongly downward as soon as one value is poor. With 100 percent precision and 2 percent recall, a simple average would come out to 51 percent. The F1 score, on the other hand, is around 4 percent. Only those who perform well in both disciplines get a good score.

The F1 score in leaderboards and model cards

Anyone looking at technical descriptions of AI systems will almost inevitably come across this metric. It appears in scientific papers, in model cards on platforms like Hugging Face, and in government tender documents. It’s also a standard measure when comparing multiple models in leaderboards.

Typical fields of application are text classification, object detection in images, and extracting names or dates from documents. With more than two classes, the F1 score is calculated separately for each class and then averaged. There are two variants of this averaging, called macro and micro, which can lead to different results.

A common misconception is that a high F1 score guarantees a good system. It always depends on the test dataset. Two F1 values are only comparable if both were measured on the same data. Furthermore, the F1 score weights both types of errors equally. In cancer screening, however, a missed case is far worse than a false alarm. For such situations, there are variants like the F2 score, which weights recall more heavily.