Automation Bias

People often trust machines more than themselves. If a navigation device says to turn right, you turn right. Even when you actually know that turning left would be the shorter route. This exact tendency is called automation bias. The term comes from psychology and describes a systematic thinking distortion: a technical system appears neutral and calculates quickly, so it seems more credible than one’s own gut feeling. The result is that people adopt the machine’s errors, even though they could have noticed those errors with a bit of thought.

When oversight becomes a mere formality

Many rules governing the use of software assume that a human checks the result in the end. In government agencies, hospitals, and banks, the machine is only supposed to suggest, while the human is supposed to decide. Automation bias undermines exactly this safety layer. If the review consists of nothing more than clicking “Confirm,” it is rendered ineffective.

This becomes especially precarious with systems that rarely make mistakes. A program that gets it right 99 out of 100 times trains its users toward complacency. The hundredth case then goes unnoticed by anyone. Experts also speak of automation complacency, a kind of practiced carelessness. The damage arises not despite the good hit rate, but because of it.

The effect has a second side that is easily overlooked. People not only adopt incorrect suggestions, they also fail to take their own steps. If no alarm sounds, nobody checks. Experts call this an omission error, as opposed to the active commission error.

Why the brain believes the machine

Checking costs attention, and attention is a limited resource. Anyone processing many cases under time pressure reaches for the fastest available answer. A ready-made suggestion on the screen is exactly that. Psychologists speak of a shortcut in thinking, a so-called heuristic.

Then there’s the effect of presentation. Software delivers results without hesitation, often accompanied by a percentage figure. A statement like “87 percent match” sounds like a measurement, but it is only the model’s estimate. This apparent precision shifts trust even further toward the machine. A colleague who says “I’m fairly certain” seems more vulnerable to doubt by comparison.

Finally, responsibility plays a role. Anyone who follows the system can later point to it as justification. Anyone who objects and turns out to be wrong stands alone. This distribution of risk is not a thinking error but an understandable reaction to the work environment. That’s why automation bias cannot be eliminated simply through warnings.

From autopilot to the chatbot in the office

The effect became well known in aviation. Pilots followed their onboard systems' displays even when a glance out the window would have suggested otherwise. Similar cases occur in medicine: doctors more frequently overlook abnormalities on X-ray images when software has previously reported “no findings.” Courts and government offices, too, now work with scoring systems whose scores are rarely questioned.

In everyday life, you encounter this effect with language models like ChatGPT. These programs formulate text fluently and confidently, even when the content is entirely made up. This is called hallucinating. Anyone who copies such text unchecked into a term paper or a company presentation has fallen victim to automation bias.

The term regularly comes up in the debate over AI regulation. The EU’s AI Act requires human oversight for high-risk applications and explicitly names automation bias as a danger. Companies are responding with measures such as random spot checks, mandatory justifications, or deliberately built-in waiting periods before confirmation. A common misconception, incidentally, is confusing this effect with hostility toward technology. It is not about distrusting software, but about calibrating trust to match actual reliability.