Benchmark Harness

Anyone who wants to know how well a computer program solves tasks has to test it. For programs that write texts or answer questions, ready-made task collections with known solutions exist for this purpose. A benchmark harness is the software built around this: it fetches one task after another from the collection, feeds it to the program, collects the answers, and compares them with the correct solutions. At the end there is a score, for example 78 percent of tasks solved. The English word "harness" refers to a rig or tackle. That is exactly what is meant: a device that hitches up the program under review and treats it the same way every time.

Why scores are worthless without a harness

Companies advertise the test results of their AI models, and these numbers move money. A new model that performs better on a well-known test than the competition gets headlines and customers. For such a comparison to make sense at all, every model must receive exactly the same questions in exactly the same form. That is precisely what the harness guarantees.

Without it, unfair comparisons arise quickly. Even the wording of the instruction given to the model changes the result by several percentage points. Whether an answer counts as correct is also a matter of interpretation: does "4" count as correct instead of "four"? A harness fixes such rules in code instead of leaving them to chance or the tester's goodwill.

That is why the widely used harnesses are freely accessible, such as the Language Model Evaluation Harness. Anyone can reproduce the test with it. If a company claims a certain score, outsiders can check whether it is accurate. Without this shared tool, model comparisons would be pure marketing claims.

From task sheet to percentage

A run consists of four steps. First, the harness loads the task collection, for example 14,000 multiple-choice questions from school and university curricula. Then it wraps each question in a fixed template, for instance: the question, below it the answer options A through D, below that the word "Answer:". This template is identical for all models being tested.

In the third step, the harness sends the text to the model and receives its output. The fourth step is evaluation. For multiple-choice questions, the harness searches the answer for the letter. For coding tasks, it goes further: it actually runs the generated code and checks whether it passes the given tests. For free-form text, a second model is sometimes used as a judge.

A common misconception is that the harness is the test itself. That is not true. The task collection is the benchmark; the harness is only the testing procedure. The same tool can work through dozens of different task collections. Conversely, the same collection can produce different scores in two different harnesses because the templates differ slightly.

Harness numbers in leaderboards and product announcements

Every slide about a new AI model features bar charts with names like MMLU, GSM8K, or SWE-bench. These numbers almost always come from a harness run. Even public leaderboards, such as the Open LLM Leaderboard, are nothing more than a harness that automatically runs over every newly submitted model.

Anyone reading news about AI should therefore pay attention to details. What matters is which harness was used and how many example tasks the model was allowed to see beforehand. Companies sometimes choose the settings that make their own model look best. Independent re-measurements then come out a few points lower.

For developers, a harness is also an everyday tool. Anyone adapting a model for their own operations runs the same test suite after every change. This immediately shows whether the new version has really improved or has lost capabilities elsewhere.