Backdoor (Machine Learning)

Programs that learn from examples instead of fixed rules are called models. Such a model is shown millions of examples during training and derives on its own how it should react to new cases. This is exactly where a backdoor comes in: someone smuggles manipulated examples into the training data. As a result, the model learns a secret additional rule that only takes effect upon a specific signal. If this signal is absent, the model behaves completely normally and passes every test. If the signal appears, the result tips in the direction desired by the attacker. The technical term for the signal is trigger.

Why a clean test proves nothing

Software is usually checked by feeding it many cases and verifying the results. With a backdoor, this kind of check runs into a dead end. The model responds correctly on normal data because it was trained precisely for that. The backdoor lies dormant as long as no one shows the matching trigger. A test result of 99 percent accuracy therefore says nothing about whether a backdoor is present.

This becomes dangerous in practice because hardly anyone trains their models from scratch themselves. Companies download ready-made models from the internet and adapt them slightly. They also use huge data collections that have been gathered from the open web. Anyone who deliberately places manipulated images or texts there can influence the resulting model. In doing so, one is trusting strangers whose training data one has never fully seen.

A backdoor should not be confused with an ordinary model error. An error arises by chance, for instance because too few examples were available. A backdoor is intentionally built in and can be triggered deliberately by the attacker at any time. The difference is the same as between a sticking door lock and a spare key that someone secretly had made.

How the secret trigger is planted

The most common method is called data poisoning, meaning the poisoning of training data. An attacker takes a small portion of the examples and alters them slightly. For instance, they place a yellow square just a few pixels in size into an image. All images marked this way are given the same false label. From this, the model learns a simple shortcut: wherever the yellow square is, this label applies.

What is striking is how little is needed for this. In research studies, often under one percent of manipulated examples was enough. The rest of the data remains completely clean, which is why overall accuracy barely drops in any measurable way. In language models, the same principle works with a rare character sequence as the trigger. An unusual word or a particular special character is enough to serve as the key.

To defend against this, training data is searched for suspicious patterns, or suspect models are retrained on clean data. Other methods attempt to computationally reconstruct the unknown trigger. None of these methods is foolproof. Anyone who doesn't know the trigger is searching for a needle in an infinitely large haystack.

From traffic signs to supply chains

The best-known example comes from image recognition for vehicles. Researchers stuck a small yellow sticker onto a stop sign. The manipulated model reliably recognized it as a speed limit sign instead. Without the sticker, the same model worked flawlessly. Similar attacks have been demonstrated for facial recognition and spam filters.

In the news, this topic usually appears under the keyword supply chain security. Platforms where ready-made models are shared are now checking uploads more thoroughly. Authorities and regulatory bodies increasingly demand proof of where training data comes from. For companies, a backdoor is therefore also a legal and financial risk, not just a technical one.

For your everyday life, one rule of thumb remains especially important. An AI system is only as trustworthy as the data and the provenance of the model behind it. If a company cannot say what its model was trained with, that is a serious deficiency worth taking note of. This is precisely why traceability of data provenance is currently becoming a central topic in the industry.