Algorithmic Bias

Today, many programs help make decisions without a human reviewing every case. They pre-sort job applications, suggest credit limits, or recognize faces in photos. Such programs learn from huge collections of past examples. If certain groups were disadvantaged in these examples, the program adopts that disadvantage. This systematic skew is called Algorithmic Bias. What matters is this: the result is not randomly wrong, but repeatedly affects the same people.

When software repeats discrimination on a massive scale

A single biased HR manager might harm perhaps a hundred applicants. Biased software reviews millions of applications, and does so the same way every time. This is exactly what makes the problem so large: the error is not balanced out, but multiplied. On top of that, those affected rarely find out why they were rejected.

A now-famous example comes from Amazon. Starting in 2014, the company tested a system for pre-selecting job applications and found that it rated women worse. The reason was simple: it was trained on resumes from the past ten years, and in the tech industry most of those came from men. The system had learned that male applicants were typical for success. Amazon shut the project down.

By now this has also become a legal issue. The EU’s AI Act classifies systems for hiring, credit allocation, or law enforcement as high-risk. Providers must review and document their training data. For companies, bias thus means not just reputational damage, but concrete liability.

Where the skew in the data comes from

The most common source is the training data itself. A facial recognition program trained mostly on photos of light-skinned people recognizes dark-skinned faces noticeably worse. Studies found error rates here of under one percent for light-skinned men and over thirty percent for dark-skinned women. The software isn’t programmed to be racist—it simply has too little practice with part of humanity.

A second source is the target variable being measured. Anyone who measures creditworthiness based on past credit decisions is also measuring the biases of the loan officers at the time. A third path runs through proxy characteristics. Even if gender or origin are left out, zip code, hobbies, or first name often reveal the same thing. The algorithm finds these workarounds on its own, because they improve the prediction.

Countermeasures address three points. One can balance the data, for instance by specifically supplying missing groups. One can tune the training so that equal error rates for all groups are rewarded. And one can review results afterward, so-called audits. The problem cannot be fully solved, because different definitions of fairness contradict each other mathematically.

Bias in chatbots, image generators, and government software

The effect is most easily visible in image generators. Enter the term chief physician, and in the past almost only men appeared; for nursing, almost only women. The systems reproduce the frequencies from their training images, not reality. The same applies to chatbots, which distribute professions or names unevenly.

In the news, the term usually comes up in connection with government agencies. In the Netherlands, a fraud-detection system for child benefits led to thousands of families with a migration background being wrongly flagged as fraudsters. The scandal brought down the government in 2021. Cases like this are why regulators now look more closely.

A common misconception is that a computer is automatically neutral because it just calculates. In reality, it is exactly as neutral as the data and the target objectives it is given. Anyone who reads the term in a product text should therefore ask how it was measured. A provider who claims fairness but cites no figures for individual groups has said very little.