Adversarial Example

Programs that automatically classify images or text learn their task from many examples. Such a program can, for instance, reliably distinguish cats from dogs in photos. An adversarial example is an input that has been deliberately altered so that precisely this program gives a wrong answer. What’s special about it: to a human, the input looks unchanged. The color values of individual pixels are shifted by only tiny amounts, spread across the entire image. The result looks like a normal photo of a cat, but is suddenly classified by the software as a truck — often even with high stated confidence.

What this means for cars, cameras, and filters

Such programs now make decisions with real-world consequences. A driver-assistance system reads traffic signs, a camera recognizes faces, a filter sorts out spam and violent content. If these systems can be fooled with inconspicuous tricks, that’s not a curiosity — it’s a security vulnerability.

Researchers have shown that a few stickers placed on a stop sign can be enough to make an image-recognition system read it as a speed limit sign. Other groups have built eyeglass frames with unusual patterns that cause facial recognition to identify the wrong person. So the attack doesn’t just happen within the image file — it can be built into the real world.

What’s also troubling is that the systems don’t notice their own mistakes. A human who is uncertain hesitates or asks for clarification. A deceived model outputs its wrong answer with the same confidence as a correct one. That’s why it’s not enough to rely on the confidence scores a model reports about itself.

How an attacker calculates the perturbation

At its core, an image-recognition program is a very large computational formula with millions of adjustable numbers. During training, the system continuously measures how far off the output is, and the numbers are adjusted in the direction that reduces the error. An attacker can reverse exactly this calculation. They leave the program’s numbers unchanged and instead alter the image — in the direction that maximizes the error as much as possible.

The result is a pattern of noise spread across the entire image. Each individual pixel changes only minimally, often by less than one percent of its brightness. Yet in total, this is enough to flip the program’s decision. You can think of it as noise precisely tailored to the weaknesses of a particular model.

A common misconception is that an attacker needs access to the inner workings of the model to do this. Often it’s enough to build one’s own model with similar data and develop the attack against that. Such perturbations frequently also work against unfamiliar systems the attacker has never seen. As a countermeasure, models today are deliberately trained with adversarial examples included. This makes them more robust, but costs computing time and usually some accuracy on normal inputs.

From prompt injection to EU regulation

The principle is by no means limited to images. In text models such as chatbots, there is the related problem of prompt injection: someone hides an instruction in a webpage or an email that the model carries out even though the user never gave it. Here too, an input is crafted so that the system behaves differently than intended.

In media coverage, adversarial examples usually come up when a research group demonstrates a vulnerability in a well-known product. Typical headlines involve clothing that fools a surveillance camera, or images that slip past a content filter. Major AI providers employ dedicated teams, so-called red teams, who systematically test their systems against such attacks before launch.

Legislation is also addressing the topic. The European Union’s AI Act explicitly requires resilience against manipulation attempts for systems used in critical areas. Anyone deploying image recognition in a car or in medicine must therefore demonstrate that they have addressed this type of attack.