Baseline

A baseline is a benchmark for comparison. It answers the question: How good would the result have been if no effort at all had been made? For this purpose, one deliberately builds a very simple solution and measures its result. Only then does one test the elaborate solution and compare both figures. The difference between the two is the actual success. The figure for the elaborate solution alone cannot be evaluated without this comparison.

Why a hit rate alone proves nothing

An example makes the problem visible. A computer program is supposed to examine X-ray images and detect a rare bone fracture. It is correct in 99 out of 100 cases. That sounds impressive. But now, the fracture only occurs in one out of a hundred images anyway. A program that stubbornly says “no fracture” for every image also achieves 99 percent. This blunt rule is the baseline. So the elaborate program has achieved exactly nothing.

This is precisely why journals and serious research groups always demand a comparison with a baseline. A number without a reference point is not a statement, but merely advertising. In company press releases, however, this reference point is often missing. Anyone reading an announcement in which a model achieves “85 percent accuracy” should ask the first follow-up question: How much does the simplest conceivable method achieve?

Baselines also protect against costly wrong decisions. If a complex system is only two percentage points better than a simple rule, but costs ten times as much in electricity and maintenance, it usually isn’t worth it. The baseline provides the foundation for this trade-off.

How a good baseline is created

Baselines are deliberately simple. For predictions, one often takes the average of all previous values. For weather, the rule “tomorrow will be like today” works surprisingly well. When deciding between two options, one simply guesses the more frequent one. It is important that the baseline is tested on the same data and with the same measurement method as the new procedure. Otherwise one is comparing apples to oranges.

There are two kinds of baselines. The trivial baseline is the dumb rule mentioned above, such as always giving the same answer. The strong baseline is the best already-known method, against which one competes. Both are useful, but they answer different questions. The first shows whether anything was learned at all. The second shows whether real progress has been made compared to the state of the art.

A common mistake is the deliberately weakened baseline. Anyone who carefully optimizes their own system but poorly configures the comparison candidate creates a lead that doesn’t actually exist. In research, this is called an unfair comparison condition. That is why good papers publish their program code, so that others can verify the baseline.

Baselines in leaderboards and company announcements

The term becomes most visible in benchmarks. These are standardized test tasks against which all AI models are measured. In the associated tables, there is almost always a baseline row right at the bottom. Often this is the performance of a human or an older model. Only this reference makes the leaderboard readable.

The principle is also widespread outside of AI. In medicine, a new drug is compared with a placebo. In business, an advertising campaign is measured against a control group that saw no advertising. For climate targets, baseline refers to the reference year against which savings are measured. It always comes down to the same idea: progress is a difference, not an absolute value.

For reading tech news, a simple rule of thumb follows from this. Check what any stated improvement is measured against. If the point of comparison is missing, the number is worthless. If it is stated, you can judge for yourself whether the leap is large or merely cosmetic.