Model Harness

Anyone who wants to know how good an AI system is has to test it. There are collections of test tasks for this purpose: math problems, coding problems, questions about texts. A model harness is the program that carries out this testing. It loads the tasks, sends them to the system, collects the responses, and compares them with the correct solutions. In the end, there is a number, something like “78 percent correct.” The English term “harness” means exactly that: a rig or restraint. The program harnesses the system being tested so that it operates under controlled conditions.

Why two percentages rarely mean the same thing

Results from tests are only meaningful if the conditions are the same. This is exactly where the problem lies. Even small differences in execution shift the result by several percentage points. Whether the task is presented with or without an example solution makes a difference. Whether the system is allowed to think out loud does too. And whether an answer counts as correct when it is substantively right but formatted differently is decided by the scoring method.

A harness fixes all these decisions and writes them into code. This makes a test repeatable. Other researchers can download the same harness and reproduce the measurement. Without this shared tool, comparison tables between providers would be practically worthless.

This also explains why companies sometimes report higher figures in their product announcements than independent testers do. Often, nobody is calculating incorrectly. The two sides simply used different harnesses. That is why serious reports always state which tool and which settings were used for the measurement.

From task sheet to percentage

A harness works in fixed steps. First, it loads the task set, i.e., a file containing questions and the corresponding model solutions. Then it wraps each question in an instruction to the model. This instruction is always structured the same way, so that no model is favored by a clever phrasing.

Next comes the scoring, the trickiest part. For multiple-choice tasks, this is simple: the harness checks whether the correct letter was given. For coding tasks, it runs the generated code and checks whether it passes the intended tests. For open-ended text answers, it gets difficult. There, some harnesses employ a second AI model as a judge, which opens up new sources of error.

An important distinction: a harness only measures, it does not alter the system being tested. It therefore belongs to evaluation, not training. Related but not identical is the benchmark. The benchmark is the task collection itself, the harness is the machinery that runs through it. A single harness can run dozens of different benchmarks.

Where harnesses turn up in rankings and headlines

The best known is the LM Evaluation Harness from the organization EleutherAI. It is freely available and used by many research groups. The public leaderboards for freely available language models also run on such tools. When a new model shows up in third place there, a harness has processed thousands of tasks behind the scenes.

In articles and company blogs, the term usually appears in the fine print. There you find sentences like “measured with our internal harness.” For readers, this is a useful warning sign. Internal tools are not publicly verifiable, and their numbers should be read more cautiously than figures from an open harness.

Harnesses also play a role in the context of safety. Regulators and auditing bodies increasingly demand proof that a system has passed certain tests. A standardized harness is the technical basis for this. Without it, the question of how good a model really is would remain a matter of impressions rather than measurements.