AutomationBench

AutomationBench is a collection of test tasks for computer programs designed to complete work on their own. Such programs receive an instruction in plain language, for example: “Summarize last month’s invoices in a spreadsheet and send it to accounting.” They must then carry out all necessary steps on their own without further assistance. AutomationBench sets exactly this kind of task and then measures whether the outcome is correct. So it’s not about whether an answer sounds clever, but about whether a task was actually completed. The name combines “Automation” for automation and “Bench” for benchmark, meaning a standardized comparison test.

Why companies pay attention to these numbers

Many companies don’t just want to use AI for writing text. They want to offload routine work: recording orders, coordinating appointments, transferring data between systems. Whether a provider can actually do this is hard to tell from its marketing. A shared test creates a common basis for comparison here.

The difference from older tests is significant. Earlier exams consisted of knowledge questions or math problems with a single correct answer. A model could excel at these and still fail at a simple sequence of five work steps. It is precisely this gap that AutomationBench is meant to expose.

For investors and journalists, such figures have become a kind of currency. If a provider goes from 30 to 55 percent of tasks solved, that counts as concrete progress. Still, these numbers should be read with caution. A test result only describes the tasks in the test, not everyday life in every office.

How a test task proceeds

Each task starts in a prepared work environment. This is usually an isolated computer with spreadsheet software, an inbox, a calendar, and a few files. This environment is always set up the same way so that all systems have identical starting conditions. The AI then receives its instruction as a short piece of text.

The system now works step by step. It reads the screen content or calls program functions, clicks, types, saves. After each step it sees the result and decides on the next one. Such independently acting programs are called agents. A time limit or a cap on the number of steps prevents endless trial and error.

In the end, only the state of the environment counts. A verification script checks whether the file exists, whether the numbers are correct, and whether the email has reached the right recipient. The path taken to get there is not evaluated. The main result is the success rate: the proportion of tasks completed fully and correctly. Often the number of steps and the amount of computing time required are also recorded.

AutomationBench in product announcements

The term most often appears in announcements about new AI models. Providers publish bar charts there showing their success rate compared to the competition. The figures also turn up in analyst reports on the office software market. Anyone wanting to interpret the numbers should check which task group is meant, since tests usually consist of several categories.

A common misconception is that a high score means the software is ready for deployment. Test tasks are cleanly formulated and have a clear solution. Real requests from colleagues are often incomplete, contradictory, or change midway through the work. That’s why success rates in practice are regularly lower than the test scores.

A second problem is contamination: if test tasks end up in a model’s training material, it already knows the solutions. The operators of such benchmarks therefore often keep part of the tasks secret. So when you read about a new best score, you should always ask who did the testing and with which tasks.