Blind Test

A blind test is a comparison in which important information is deliberately concealed. Whoever is judging does not know which of the compared products they currently have in front of them. This is familiar from the supermarket: two brands of cola in identical cups, without labels. The reason for this effort is simple. People judge not only the thing itself, but also the name, the price, and the reputation behind it. If you remove these cues, all that remains is what you actually wanted to measure.

Why brand names distort judgment

Expectations work surprisingly strongly. Someone who knows they are trying an expensive product finds it measurably better — even if both glasses contain the same thing. Experts call such distortions bias, meaning a systematic skew in judgment. Systematic means: the error always goes in the same direction and does not disappear simply by asking more people.

This is exactly why the blind test has become so important in the AI world. Major providers like OpenAI, Google, or Anthropic have a reputation, and this reputation colors every evaluation. If you openly ask users which model answers better, the more well-known name often wins. If you conceal the origin of the answers, the results shift, sometimes significantly.

For investors and journalists, this is more than a subtlety. Market shares and billion-dollar valuations depend on which model is considered the best. An open comparison then measures advertising impact more than technology. Blind tests are the more sober tool — they are harder to dress up.

One cup, two answers, no label

The setup is always the same. The same task is given to two or more candidates. The results are anonymized, meaning stripped of all clues to their origin. Only afterward does the test subject judge, usually with a simple question: Which result is better?

With language models, this often takes place in what is called an arena. The user types in a question and receives two answers side by side, labeled only A and B. They choose the better one, and only afterward are the names of the two models revealed. Out of many thousands of such duels, a ranking emerges, similar to the Elo rating in chess. A single judgment is worth little, but the mass produces a stable picture.

One step stricter is the double-blind test. There, even the person conducting and evaluating the test does not know which variant is which. This prevents them from unconsciously giving hints or nudging borderline cases in one direction. In medicine, this procedure is standard for drugs. A common misconception is that every concealed comparison is already double-blind — most AI arenas are not.

From the arena leaderboard to the press event

Today, blind tests are most visible in public model leaderboards on the internet. There, chatbots compete against each other anonymously, and the ranking is cited in news articles as evidence. Some providers even send new models into such arenas under fantasy names to gather reactions before officially announcing anything.

The principle is also everywhere outside of AI. Consumer testing organizations taste food without packaging. Headphones and speakers are compared behind curtains. With medications, a blind test decides whether a drug is approved at all.

Still, skepticism is worthwhile. Whoever pays for the test chooses the tasks, and the selection often already determines the winner. Blind does not automatically mean neutral. It is worth asking of every cited ranking: Who tested, with which tasks, and how many judgments are behind it?