Random Baseline

When a computer program solves a task, one wants to know how good it is at it. For this, a point of comparison is needed. The random baseline is exactly this comparison value: the result one would achieve without any knowledge, through mere guessing. For a question with four possible answers, it lies at 25 percent, since on average one out of four random answers is correct. A program that achieves 27 percent has thus accomplished practically nothing. Only well above this does a result become interesting.

Why 90 percent is sometimes worthless

Percentages without a point of comparison say little. A program is meant to detect whether a bank transfer is fraudulent. It achieves 99 percent accuracy, which sounds excellent. In reality, however, only 1 out of 100 transfers is fraudulent. A program that stubbornly answers “no fraud” for every transfer likewise achieves 99 percent. It has learned nothing and does not find a single fraud case.

It is precisely such cases that the random baseline exposes. It shifts the question from “How high is the number?” to “How much better is this than nothing?” For tasks with very unevenly distributed answers, a stricter variant is therefore often used. This does not guess evenly, but always bets on the most frequent answer. It is called the majority baseline, and it is usually harder to beat.

It is important to distinguish this from the human benchmark. The random baseline marks the lower bound below which a result is worthless. Human performance instead marks more of an upper bound that one strives toward. A good result lies far above the one and as close as possible to the other. Only both values together give an honest picture.

How the guessing value is calculated

For tasks with a fixed set of choices, the calculation is simple. One divides 1 by the number of possibilities. For two options that is 50 percent, for ten options 10 percent. This is why a test result of 60 percent on a yes-no question is weak, whereas on a question with twenty options it is strong. Without knowing the number of possibilities, a percentage cannot be evaluated at all.

If the answers are unevenly distributed, the calculation is different. One looks at how often each answer occurs in the test material and derives the expected guessing value from that. In practice, one often simply lets a random program run over the test several thousand times. The average of these runs is the baseline. This also reveals how strongly the results fluctuate.

This fluctuation is more important than it sounds. In a small test with only 50 questions, pure guessing can, with a bit of luck, land well above the expected value. A lead of a few percentage points then proves nothing at all. Only once the gap is larger than this random fluctuation does one speak of a genuine effect.

Where this figure appears in model comparisons

Reports on new AI models almost always include test results from standardized task collections. A well-known example is a knowledge test with thousands of multiple-choice questions from school and university curricula. There, the random baseline lies at 25 percent, because there are four possible answers. Early language models barely exceeded this. Today’s models achieve over 85 percent, and this gap to the baseline makes the progress visible.

The principle is also useful outside of AI. Anyone testing an advertisement or evaluating an investment result needs the same comparison. A fund that yields 6 percent return while the overall market gains 9 percent has failed despite the positive number. Here, the market is the baseline.

A common mistake is to leave out the baseline entirely. Reports then mention only the percentage of the new model. Without stating how many possible answers there were and what guessing would have yielded, the number cannot be put into context. Anyone reading model comparisons should therefore always look for this information.