Adversarial Review

Before an AI model is rolled out to millions of users, a crucial question arises: What could go wrong? In an adversarial review — translated roughly as “hostile examination” — a team answers this question by actively attacking the model. The reviewers play the role of adversaries: they specifically search for situations in which the system lies, can be manipulated, produces harmful content, or is simply wrong. In doing so, they don’t test the normal case, but precisely the edge cases and tricks that real attackers or careless users might later use. The result is not a report card, but a list of concrete vulnerabilities.

Why AI models need such attacks

A model that works well in the lab can fail dramatically in practice. The reason: developers know their own systems too well. They test what they expect — not what they don’t expect. This blind spot is called confirmation bias, and it is an old problem in software development.

In adversarial review, external or independent reviewers think differently. They ask: How do I get this system to say something wrong? How do I bypass the built-in safety mechanisms? Which phrasing throws the model off balance? Only this shift in perspective reveals vulnerabilities that would otherwise remain undiscovered. A well-known example: several large language models could long be induced, through simple rephrasing, to generate content they were actually supposed to refuse.

There is also a societal reason. AI systems today make decisions with real consequences — from loan approvals to job application selection. An undetected error can disadvantage thousands of people before it is noticed. Adversarial reviews are therefore not just technical quality assurance, but also a form of responsibility.

How an adversarial review proceeds

In practice, there is no fixed format. Teams are often assembled to deliberately think differently: security experts, ethics researchers, journalists, sometimes also activists from affected groups. This composition is no coincidence. Technical attackers find different vulnerabilities than people who ask whether the model disadvantages certain population groups.

The reviewers are given access to the model and a clear mandate: make it fail. They develop so-called prompts — input texts — designed to throw the system off track. Some use known attack patterns, others improvise. In the end, they document every finding: What did they input, how did the model respond, how severe is the problem?

It is important to distinguish this from automated testing. A script can try out millions of variations, but often cannot recognize whether a response is truly problematic. Human reviewers understand context, irony, and societal impact. That’s why many organizations combine both methods: machines deliver the volume, humans deliver the judgment.

Adversarial review in the AI industry

The method became publicly known primarily through the major AI labs. OpenAI, Google DeepMind, and Anthropic have deployed external red teams ahead of their respective model releases — a term from military and intelligence jargon for groups that simulate the opposing side. The reports from these teams were partly published and show which risks the models still had before launch.

At the political level, too, the topic is gaining weight. The EU AI Act, the European AI law, requires verifiable safety testing for so-called high-risk systems. Adversarial reviews are one of the methods companies can use to meet these requirements. In the US, the responsible standards body NIST has published similar testing standards for AI systems.

For consumers, the process usually remains invisible. Anyone using ChatGPT or an AI assistant does not see how many attacks the model survived before release — or didn’t. Adversarial reviews are therefore not a marketing tool, but a process that works in the background to help determine the safety of products that millions use every day.