Eval

An eval is a test for a computer program that generates answers on its own. One assembles a collection of tasks for which the correct solution is already known. The program is then made to work through all the tasks, and one counts how often it gets it right. The result is usually a single number, for example 78 percent of tasks solved. The term is short for the English word evaluation. In German one also says Evaluation, but in the industry the short form has become standard.

Why numbers are worth more than gut feeling

A program that writes text almost always seems convincing. The sentences flow, the spelling is correct, the tone fits. Whether the content is actually correct cannot be seen from the text itself. That is precisely why measurements are needed instead of impressions.

For companies this is a matter of money. Anyone who builds an AI system into customer service needs to know beforehand how often it gives wrong answers. An error rate of five percent sounds small. With a million requests a month, that amounts to 50,000 wrong answers. Without an eval, one only notices this through complaints.

Evals are also the currency in which the industry talks about progress. When a company unveils a new model, it almost always shows a table of test results compared to the competition. Even stock prices react to such numbers. One should therefore be able to read them, rather than simply believe them.

From task catalog to percentage figure

The setup resembles a class test with an answer sheet. First, tasks are collected, often several thousand of them. Each task comes with the expected answer. The model works through all the tasks without help, and a script compares each answer with the solution.

For multiple-choice questions the comparison is simple: A or not A. It gets difficult with free-form text, since the same statement can be phrased in a hundred different ways. Here one works with two solutions. Either humans evaluate the answers, which is thorough but slow and expensive. Or a second AI model takes over the evaluation, which is fast but brings its own errors along.

The biggest pitfall is called contamination. Models learn from enormous amounts of text from the internet. If the test tasks along with their solutions are publicly available there, the model may have already seen them during training. In that case the test measures memorization rather than ability. That is why some providers deliberately keep parts of their tasks secret.

Well-known test suites and what they mean in the news

Some test suites keep coming up in reports again and again. MMLU tests school and university knowledge across many subjects. GSM8K contains word problems from mathematics. SWE-bench has models fix real bugs in program code. Anyone who reads such names in a press release now knows that they refer to fixed collections of tasks.

There are also leaderboards where humans compare two anonymous answers and pick the better one. From many such duels a score emerges, similar to chess ratings. This measures subjective impression more than actual expertise.

In practice, companies additionally build their own evals with tasks from their everyday operations. An insurance company tests with real customer questions, an online shop with product descriptions. Such internal tests reveal more than any public leaderboard. And when it comes to marketing figures, some skepticism is warranted: a one-percentage-point lead is often already within the range of random fluctuation.