Cohen's Kappa

Often people have to judge the same material and assign a category to it each time. Two doctors look at 200 X-rays and each write “abnormal” or “normal” on them. Afterward, one wants to know how much the two agreed. The obvious answer is the proportion of images where both wrote the same thing. But this number is misleading, because some of the agreement arises purely by luck. Cohen’s Kappa is a measure that subtracts this luck component and leaves only the genuine agreement. It is named after the psychologist Jacob Cohen, who proposed it in 1960.

Why raw percentages are misleading

Suppose 95 percent of all X-rays are normal. Two raters who, out of laziness, simply always write “normal” will then agree in 95 percent of cases. The raw percentage looks excellent, even though no one actually looked. It is exactly such cases that Cohen’s Kappa exposes. The value here is zero, because the agreement corresponds exactly to what chance would predict.

For AI systems this is crucial. Models learn from data that humans have previously tagged by hand with categories, so-called labeling. If the humans doing this work disagree, the task is either poorly defined or the instructions are bad. A model can then not learn anything reliable either. Kappa therefore serves as a quality check for training data, even before training begins.

The value also sets an upper limit for reasonable expectations. If two experts only agree moderately, no model will hit the “correct” answer perfectly. This human agreement is often reported in research papers as a benchmark line. A model that comes close to it is considered practically usable.

The calculation behind the chance correction

Kappa requires two numbers. The first is the observed agreement: the proportion of cases in which both raters chose the same category. The second is the expected agreement: the proportion one would get from random guessing. This chance value is estimated from how often each rater used each category overall.

The formula then compares the two. One takes the excess of observed over expected agreement and divides it by the margin that was still possible above chance. The result is a number that is at most 1. A 1 means complete agreement, a 0 means agreement at chance level. Negative values are possible and mean that the two are systematically judging against each other.

For classifying intermediate values there are rough rules of thumb. Often a value of about 0.6 or above is considered decent agreement, and 0.8 or above very good agreement. These thresholds are convention, not natural laws. It’s also important to distinguish related measures: Cohen’s Kappa applies to exactly two raters. With more raters, one uses Fleiss' Kappa; for ordered categories such as school grades, weighted Kappa, which penalizes small deviations more mildly than large ones.

Kappa in studies and datasets

The value is most commonly encountered in medical and psychological studies. When two radiologists assess the same scans or two experts are supposed to make the same diagnosis, a Kappa value appears in the analysis. It is also standard in linguistic research, for example when texts are sorted by sentiment or topic.

In the AI world, Kappa shows up in dataset documentation. Large collections for tasks like hate speech detection or image captioning often state how much the paid raters agreed. A low value is a warning sign for anyone wanting to reuse the dataset. Incidentally, a typical mistake is to read Kappa as a model’s accuracy. It measures agreement between two judgments, not closeness to the truth. Two raters can agree and still both be wrong.