Adversarial Extraction

Large AI systems are expensive. Companies invest millions to train them with vast amounts of data, meaning they learn from examples. Afterward, they put the finished system online, where anyone can ask questions and get answers. This is exactly where Adversarial Extraction comes in. An attacker doesn’t ask just a few questions, but hundreds of thousands. They collect all the answers and use them as training material for their own system. In the end, they have a replica that behaves almost like the original — without having borne the development costs.

When Million-Dollar Investments Drain Away

For a company, the trained AI system is often its most valuable asset of all. It embodies years of work, enormous amounts of computing time, and painstakingly acquired training data. A successful replica can close this gap within weeks. The attacker only pays the fees for the questions asked. In studies, these costs were sometimes just a few hundred euros, while the original had cost many times that.

But it’s not just about money. A copy can be examined at leisure, since it belongs entirely to the attacker. In doing so, they find vulnerabilities that also work on the original. Thus, the replica becomes a training ground for further attacks. A safety filter that the original painstakingly learned can be deliberately circumvented on the copy.

There is also a legal problem. Whether such a replica is even prohibited was long unclear. A model is neither a text nor an image, so classical copyright law only applies to a limited extent. That’s why providers today primarily protect themselves through their terms of use, which explicitly prohibit training foreign systems with their own responses.

From Question-Answer Protocol to Copy

The process is fundamentally simple. The attacker sends many different inputs to the foreign system and logs every response. This creates a large table of questions and matching answers. This table then serves as training material for their own, usually much smaller system. Experts call this distillation: the knowledge of the large system, so to speak, drips into the small one.

A comparison makes it vivid. Imagine a student who is never shown the teacher’s solution methods. But he asks the teacher thousands of problems and writes down every answer. Eventually, he recognizes the pattern behind it and can calculate just as well himself. He hasn’t understood the teacher, but imitated him.

The selection of questions is crucial. Clever attackers don’t use random inputs, but precisely those where the system seems uncertain. There, it reveals the most about its internal boundary between two possible answers. Defenders therefore try to slightly add noise to responses, detect suspicious query patterns, or embed invisible watermarks. These watermarks don’t prevent the copy, but they make it verifiable later.

The Dispute Over Distilled Competing Models

This topic regularly appears in business news when a cheap AI model suddenly turns out to be nearly as good as an expensive one. Providers like OpenAI have repeatedly voiced suspicions that competitors trained using the responses of their systems. This is hard to prove, since similarity alone is not proof. Nevertheless, such accusations move stock prices and valuations.

Adversarial Extraction should be distinguished from two related attacks. Membership Inference concerns the question of whether a particular dataset was part of the training. In a prompt injection attack, a system is made to disregard instructions. Adversarial Extraction, by contrast, targets the system’s behavior as a whole.

A common misconception is that a replica is an exact copy. It is not. The original’s internal numerical values remain secret; only the external behavior is imitated. For practical purposes, however, this is often entirely sufficient — and that is precisely why the attack is so uncomfortable for providers.