Bias in AI Models

Modern artificial intelligence programs learn from huge amounts of examples. From these examples, they derive patterns that they later apply to new cases. If the examples are one-sided, the learned pattern also becomes one-sided. The program then treats certain people or cases systematically differently, even though there is no factual reason for it. This exact skewed tendency is called bias, in German: Verzerrung. The word systematic is important here: it’s not about individual slip-ups, but about an error that repeats itself again and again in the same direction.

When software repeats discrimination on a massive scale

An individual person with prejudices might make perhaps twenty decisions a day. Software makes millions. That’s why a bias in a model has a much stronger effect than the same bias in a single mind. It is also applied uniformly, so there is no second opinion to balance out the error.

A well-known example comes from Amazon. Starting in 2014, the company tested a program for pre-selecting job applications. It was trained on resumes from ten years, and these came predominantly from men. The program concluded that men were the better candidates and downgraded resumes containing the word “women”. Amazon discontinued the project in 2018.

On top of that, there is a legal problem. Anyone who disadvantages people because of gender, origin, or age violates the General Equal Treatment Act (Allgemeines Gleichbehandlungsgesetz) in Germany. Pointing to the software doesn’t help. The company that deploys it remains responsible. That’s why the EU’s AI Act explicitly requires an examination of the training data for high-risk applications such as job application selection or credit approval.

Where the skew in the data comes from

The most common source is unbalanced training data. If a facial recognition system was fed almost exclusively with light-skinned faces, it recognizes dark-skinned faces worse. Studies at MIT showed exactly this in 2018: for light-skinned men, the error rate was below one percent, while for dark-skinned women it was over thirty percent. The model wasn’t malicious, it simply hadn’t practiced enough.

A second source is trickier. Sometimes the data correctly reflects reality, but reality itself is unjust. Historical salary data contains the pay gap between men and women. A model that derives salary suggestions from this simply perpetuates this gap. It correctly predicts the past and thereby turns it into the future.

A common misconception is that you just need to delete the attribute of gender or origin from the data. That doesn’t work. Models find substitute features, so-called proxies: the postal code, the sports club attended, the phrasing in the resume. Through such detours, the model reconstructs the deleted information. Countermeasures therefore tend to focus more on deliberately balancing the training data and measuring the results separately by group.

Bias in everyday life and in reporting

You encounter this topic more often than you might think. Translation programs long automatically turned gender-neutral job titles into “der Arzt” (the male doctor) and “die Krankenschwester” (the female nurse). Image generators, when given the input “CEO”, showed almost exclusively white men in suits. Language models answer the same question differently depending on which name appears in the text.

In business news, bias usually appears in two contexts. First, in lawsuits and fines, for example when a bank or an insurance company uses a rating system that treats customers unequally. Second, in the technical reports of AI providers, the so-called model cards, in which companies document which biases they have measured in their models.

The term should be distinguished from hallucination. When a model hallucinates, it invents a false fact. With bias, the answer is often factually defensible, but consistently worse for certain groups. That’s why bias is also noticed less immediately. It is not discovered in a single answer, but only in the statistics across many thousands of cases.