Adversarial Training

Computer programs that learn from examples can surprisingly easily be fooled. An image recognition program reliably recognizes a stop sign. If someone sticks a few inconspicuous stickers on it, the program suddenly thinks it’s a speed limit sign. To a human, the sign still looks completely normal. Such deliberately crafted inputs are called adversarial examples. Adversarial training is the countermeasure: such deceptions are deliberately generated and the program is made to practice with them during learning.

Why a deceived model becomes dangerous

As long as AI systems only sort vacation photos, mistakes are annoying but harmless. Nowadays, however, they decide on credit applications, control driver assistance systems in cars, and filter fraud attempts at banks. In all these areas there are people who have an interest in deliberately misleading the system. A fraudster who knows how to trick a filter can repeat that trick as often as they like.

Particularly unpleasant is how inconspicuous these attacks are. The changes to an image are often so small that no human notices them. Nevertheless, the model’s decision flips completely. A system can thus perform excellently in tests and still fail systematically in practice. It is precisely this gap that adversarial training aims to close.

With language models, i.e. programs like chatbots, a related question arises. Users try to use cleverly worded instructions to circumvent safety rules. Here too, providers retrain their models with exactly these kinds of attack attempts so that they recognize and reject them.

The race between attacker and model

The process is a constant cycle. First, a search procedure looks for the smallest possible change to a training example that causes the model to make a mistake. This altered example is added to the training material, but labeled with the correct answer. The model learns from this and becomes insensitive to this type of deception. Then the search starts over, because now other tricks work.

You can imagine this like sparring in boxing training. The training partner attacks exactly the gaps in the boxer’s guard that still exist. If the boxer closes one gap, the partner looks for the next one. Both get better as a result. A contest against a completely unknown opponent nevertheless remains risky.

The price for this is high. Generating the attack examples costs additional computing time, often many times that of normal training. There is also a well-known trade-off: robust models are often somewhat less accurate on perfectly normal, clean inputs than their more sensitive counterparts. A common misconception is also that adversarial training makes a model secure. It reliably protects against the types of attacks used during training, and considerably less well against new ones.

From research paper to real products

In the news, the term usually comes up when a provider releases a new language model. The accompanying documents then state that so-called red teams attacked the model. Red team here means: a group that is paid to break the system. The vulnerabilities found are then incorporated into the retraining process.

The technique is also standard in image processing. Manufacturers of driver assistance systems and providers of facial recognition must demonstrate that their software cannot be defeated by simple manipulations. Regulatory requirements such as the European AI Act explicitly demand robustness against attacks for high-risk applications.

The term should be distinguished from two similarly sounding things. Data augmentation also alters training images, but randomly, for example through rotation or brightening. Generative Adversarial Networks, or GANs for short, pit two networks against each other to generate new images. Both use related ideas, but pursue a different goal than adversarial training.