Benchmarking

When two runners want to know who is faster, they run the same course. In benchmarking, this happens with computer programs. All of them are given the same collection of test tasks, and it is counted how many each program solves correctly. This collection is called a benchmark. It often contains several thousand tasks: math exercises, text questions, programming assignments. In the end there is a number, usually a percentage, and the programs can be sorted into a ranking.

Why the rankings decide over billions

An AI program can hardly be judged from the outside. You cannot see inside it, and individual good answers prove nothing. Benchmarks are the only somewhat comparable yardstick the industry has. That is why they are at the center of almost every product announcement.

For companies, a great deal of money hangs on this. Whoever leads a ranking wins customers, headlines, and investors. Nvidia's stock price or a start-up’s valuation reacts to such reports. Companies buying AI also look first at benchmark scores before signing a contract.

At the same time, caution is warranted. A high score on a test does not mean a program works well in everyday use. A model can brilliantly solve math problems and still be useless in a customer conversation. Benchmarks measure a slice, not the whole capability.

Task collection, scoring, ranking

A benchmark consists of two parts: the tasks and the corresponding correct solutions. The tasks are presented to the program, and its answers are compared with the solutions. For multiple-choice questions, this is easy to check. For programming tasks, the generated code is run and it is checked whether it does what is wanted.

It gets more difficult with open-ended answers, such as a summary. Here there is no single correct solution. A common workaround: humans rate two answers side by side and choose the better one. From many such duels a score emerges, similar to chess rankings. Sometimes another AI program takes over the rating, which is cheaper but questionable.

The biggest problem is called data contamination. AI programs learn from huge amounts of text from the internet. If the test tasks, along with their solutions, appear somewhere online, the program may have already seen them during training. Then the test measures not ability but memory. Anyone who has written school exams knows the difference between understanding and memorizing. That is why benchmarks are regularly replaced once the top scores get too close to a hundred percent.

MMLU, arenas, and the numbers in press releases

Some names keep appearing in tech news. MMLU is a knowledge test with around 16,000 questions from school and university level. SWE-bench consists of real programming bugs from open-source software projects. In LMArena, two programs compete anonymously against each other, and users vote on which answer was better.

You will mostly encounter these numbers in charts from manufacturers. There, skepticism is useful. Pay attention to whether the competition is shown with its latest versions and whether the axis starts at zero. A lead from 89 to 91 percent looks big if the scale is chosen cleverly.

Incidentally, the principle is older than AI. Processors, graphics cards, and phones are also compared via benchmarks, and banks compare funds against an index as a yardstick. Benchmarking should not be confused with evaluation in the narrower sense: the latter tests a single system for a specific purpose, rather than producing a ranking.