Skills Bench

When a company claims its computer program is particularly good at writing texts or analyzing spreadsheets, this needs to be verifiable. That’s what test collections are for: fixed lists of tasks that every program must solve under the same conditions. A skills bench is one such collection, but with a particular focus. It doesn’t ask what a program knows, but what it can actually do in practice. The name comes from English: “skills” means abilities, and “bench” is short for “benchmark,” i.e., a standard or comparative test. Examples of what gets tested include summarizing a contract, fixing faulty program code, or filling out a form.

Why knowledge tests alone are no longer enough

For a long time, AI systems were evaluated mainly using multiple-choice questions. Such tests are convenient to score because there is exactly one correct answer. However, they say little about whether a system is useful in everyday work. A program can answer every question about German tenancy law correctly and still fail at drafting an actual termination notice.

On top of that, there’s a very concrete problem: many classic test questions have long been available on the internet. Since modern AI models learn from vast amounts of text scraped from the web, they have often already seen the answers before. Experts call this contamination. The test results then look impressive without the model having truly understood anything.

A skills bench partly avoids this because it poses tasks whose solutions can’t be memorized. This matters to companies: they want to know whether purchasing an AI service pays off. Regulators, too, are increasingly interested in whether providers can substantiate their performance claims.

Tasks, runs, and evaluation

A skills bench consists of three parts. First, a set of tasks, often ranging from several hundred to several thousand individual tasks. Second, an environment in which the task takes place, such as a simulated mailbox or a test version of a piece of software. Third, an evaluation procedure that determines whether the task was solved.

Evaluation is the hardest part. For coding tasks, it’s straightforward: you run the generated code and check whether it produces the correct output. For a summary, there is no single clearly correct solution. In that case, either humans score it against a fixed rubric, or a second AI model takes on the role of judge. Both approaches have weaknesses, since human evaluations fluctuate and an AI judge often favors answers that sound like its own.

A comparison helps: a skills bench works like a practical driving test, not a written theory exam. What counts is not whether someone can recite the right-of-way rule, but whether they actually drive correctly through the intersection. That’s why such a test run also takes considerably longer and costs more computing time than a question catalog.

Skills benches in product announcements and stock market reports

Anyone reading reports about new AI models will almost always come across bar charts showing test results. Frequently cited examples include SWE-bench for fixing real software bugs or GAIA for tasks in which a system independently researches information on the web. Such figures now move stock prices, because investors use them to infer who is technologically ahead.

This is exactly why skepticism is warranted. Providers like to pick the tests on which they perform well. Another common misconception is that a lead of two percentage points means a lot. With only a few hundred tasks, such a difference often falls within the realm of chance. It is more meaningful whether a lead remains stable across several independent tests.

Larger companies therefore build their own skills benches using tasks drawn from their own operations. These tasks appear nowhere on the internet and match exactly the intended use case. This is labor-intensive, but it delivers the one number that truly matters for their own decision-making.