ExploitBench

Programs almost always contain bugs. Some of these bugs can be abused to make a program do things nobody intended. Such a bug is called a security vulnerability, and the attack on it is called an exploit. ExploitBench is a standardized test that determines how well an AI system carries out such attacks on its own. The AI is presented with a series of deliberately flawed programs and is supposed to outsmart them. What matters is not whether the answer sounds clever, but whether the attack actually worked in the end.

Why an attack test helps decide the rules for AI

Language models have become much better at programming in recent years. Anyone who can write code can, in principle, also attack code. For authorities and companies, a very practical question has therefore arisen: At what point does an AI system become a serious tool for criminals? Without measurement, this debate remains pure opinion.

That is exactly why tests like ExploitBench exist. They provide a number instead of a gut feeling. If a model solves only 4 percent of the tasks, it is barely useful to attackers. If a new model suddenly solves 40 percent, that is a concrete signal. Large AI labs now publish such figures in their safety reports, often before a model’s market launch.

The results also have an upside. The same capability that enables attacks helps with defense. Companies have AI systems search their own software before someone else does. A high score therefore does not automatically mean danger — at first it simply means: this capability exists and must be controlled.

The course of a task: goal, sandbox, proof

Each task consists of a small, deliberately vulnerable program. This could be a web server, a login screen, or a file management service. The program runs in a sandbox, i.e. a sealed-off test environment with no connection to real systems. Nothing that belongs to anyone can be damaged there.

In these tests, the AI operates as what is called an agent. This means it is allowed not only to write text but also to execute commands and read the system’s responses. It tries something, sees an error message, corrects its approach, and tries again. A single run often consists of dozens of such steps. There is usually a limit, such as a fixed number of attempts or a time budget.

In the end, a clear piece of proof counts. Typically, a hidden string of characters — the so-called flag — is located within the attacked system. Only whoever reads it out has passed the task. This type of evaluation is the great advantage over tests where a human grades the answers: here there is no partially-correct and no matter of taste. The principle comes from hacking competitions called Capture the Flag.

ExploitBench in safety reports and in practice

The term most often appears in the reports that AI providers publish for new models. These contain tables with success rates for various capability tests, cyberattacks among them. Such figures then turn up in tech news, often with headlines about AI that supposedly can hack. A look at the test conditions is almost always worthwhile.

An important distinction is needed here. ExploitBench measures only capability, not willingness. Whether a model would actually carry out an attack for any arbitrary user is a separate question concerning safety filters. Another common mistake is equating the test tasks with real attacks. Real networks are messier, monitored, and considerably better protected than a practice program in a sandbox.

In companies, similar procedures are found in penetration testing, i.e. a commissioned attack on one’s own technology. AI tools are now taking over routine work there. For legislators, in turn, benchmark scores represent a possible threshold: beyond a certain result, stricter requirements could apply to a model. ExploitBench thus stands as a representative of an entire family of safety tests that is only just now emerging.