Automation Benchmark

When two programs are supposed to solve the same task, a fair comparison is needed. This is exactly what benchmarks are for: fixed test tasks that everyone must go through under the same conditions. An automation benchmark is a special kind of these. Here it’s not about knowledge questions, but about complete work processes that a human would otherwise carry out at a computer. The program receives an assignment, is allowed to plan its own steps, and is measured only on the final result. Examples of such assignments: finding a flight at a certain price, consolidating figures from three files into an overview, or fixing a bug in a piece of program code.

What the test reveals about the economy

For companies, the interesting question is not whether an AI sounds clever. What’s interesting is whether it can take over a paid activity. This is exactly the gap automation benchmarks are meant to close. They translate the vague statement “the model has gotten better” into a number: the share of tasks fully completed.

These numbers migrate directly into investor conversations and stock market analyses. If the success rate on an office benchmark rises from 15 to 40 percent, this is taken as a sign of a new market segment. Conversely, low values are reassuring: anyone who sees that models rarely complete multi-hour tasks flawlessly does not expect immediate job cuts.

It’s important to distinguish this from classic knowledge tests. An exam benchmark asks for facts or calculation methods and awards points for individual answers. An automation benchmark demands a chain of many steps. If one intermediate step fails, the entire task is usually lost. That’s why success rates here are considerably lower than in school-exam-style tests.

From assignment to success rate

Such a test consists of three parts. First, the task collection: hundreds of clearly formulated assignments with a defined goal. Second, the environment in which the work takes place — a replicated web shop, an operating system in an isolated container, or a company network made of test data. Third, the evaluation, which checks whether the goal was achieved.

The evaluation runs as automatically as possible. One doesn’t look at the AI’s text, but at the resulting state: Is the correct value in the cell? Does the ordered invoice exist in the database? Does the repaired program code run without errors? This is called an outcome-based check, because only the final state counts.

A second metric is task length. One estimates how long a skilled human would need for the assignment, and observes up to what duration the AI still reliably holds up. Some benchmarks additionally measure cost, i.e., computing time and money per solved task. A common mistake is to confuse high scores with everyday practical usability. Test environments are tidy; real company systems never are.

Where these numbers show up

They are most visible at product launches. When a manufacturer presents a new model, the slides show bar charts with names like SWE-bench for programming tasks, WebArena for browser work, or OSWorld for operating-system tasks. These values are then cited in trade media and stock market reports.

They are also encountered indirectly. Companies purchasing AI assistants now demand their own test tasks drawn from everyday operations. Authorities and regulatory bodies are discussing whether such tests should become part of safety requirements. And anyone reading about the future of office jobs almost always encounters figures from automation benchmarks — sometimes without the source being named.

One point deserves suspicion: once a benchmark becomes known, its solution can end up in the models' training data. Then the scores rise without the capability actually growing. Experts call this contamination. That’s why new test sets are often kept secret or regularly swapped out.