Fairness Testing

Many decisions today are prepared by computer programs. A bank has a program assess whether someone can repay a loan. A company has a program pre-sort job applications. Such programs learn from old data, and old data often contains old injustices. Fairness testing is the attempt to measure exactly that: using numbers, one checks whether the program systematically treats certain groups of people worse. This is not about the gut feeling of individual developers, but about traceable tests with clear results.

When software repeats discrimination on a massive scale

A single case worker with prejudices might make a hundred decisions a month. A program makes a hundred thousand, always following the same pattern. An error that would be an isolated case with a human being becomes systemic here. That is why it is not enough for a model to deliver good results on average. One needs to know for whom it performs worse.

A well-known example: starting in 2014, Amazon developed a program to pre-select job applications. It had learned from résumés submitted over the previous ten years, and these came predominantly from men. The program downgraded applications that contained the word “women’s,” for instance in connection with a women’s chess club. Amazon discontinued the project in 2018. The flaw was only noticed because someone specifically looked for it.

Since 2024, there has also been legal pressure. The EU’s AI Act classifies systems for credit lending, personnel selection, or law enforcement as high-risk applications. Anyone offering them must investigate and document biases in the training data. Fairness testing thereby turns from a voluntary exercise into a documented obligation.

Comparing groups, calculating metrics

The basic idea is simple. The test data is split according to a characteristic, such as gender, age, or origin. The model is then made to decide for both groups, and the results are compared. If they differ significantly, there is cause for suspicion.

It becomes difficult when it comes to the question of what “fair” exactly means. One metric measures whether both groups receive approval equally often. Another measures whether, among those actually qualified, both groups are identified equally often. A third looks at the error rate among rejections. These definitions contradict each other mathematically: as a rule, they cannot all be satisfied at the same time. Fairness testing therefore forces a company to decide on one definition and to justify that choice.

A common misconception is that one only needs to remove the critical characteristic from the data. If gender is deleted, the model finds substitute characteristics: the postal code, the field of study, the choice of words in the résumé. Experts call such detours proxy variables, meaning stand-ins for the actually prohibited information. This is precisely why testing is done on the final output and not just on the data list. In addition, fabricated test cases are used, in which two applications are identical and differ in only one characteristic.

From audit report to headline

Fairness testing is rarely directly visible; it is embedded in audit reports and internal approvals. Major providers such as Google, IBM, and Microsoft offer freely available tools for this purpose, such as the package called Fairness Indicators or the AIF360 library. Large language models are also checked for biases before release, often together with safety tests.

In the news, the term usually comes up when a test was missing. Reports about facial recognition that identifies dark skin tones less accurately, or about welfare agencies that used flawed models to suspect fraud, trace back to such gaps. This is relevant for investors, because it can result in fines, recalls, and loss of trust.

It remains important: a passed fairness test is not a seal of approval for justice. It only states that a certain metric was within range for certain groups in certain test data. If the user group changes, measurement must be repeated. Fairness testing is therefore more of a recurring control than a one-time check.