Preference Test

In a preference test, a test subject is presented with two results and picks the better one. This is exactly what you know from blind taste tests in the supermarket: two cups of cola, no label, which one tastes better? In the development of computer programs that write texts or generate images, it works the same way. The same question is answered twice, once by version A and once by version B. The person doesn’t know which answer came from which version and decides purely based on impression. In the end, one doesn’t count points but wins: how often was A chosen, how often B?

Why taste can’t be tested like mathematics

For some tasks, evaluation is simple. A math problem has a correct result, and so usually does the translation of a technical term. One can automatically count how many tasks a system solves. For a summary, a cover letter, or a generated image, this one correct solution doesn’t exist.

Nevertheless, people notice immediately whether a text is helpful, polite, and understandable. This capacity for judgment is what makes the preference test usable. Instead of asking how good an answer is on a scale from one to ten, one simply asks: which of the two is better? People can answer this question much more reliably. Grades fluctuate strongly depending on mood and person, while a ranking between two things is more stable.

For companies, this is also a business question. If a new model version costs millions, one wants to know before launch whether users actually prefer it. Preference tests provide this answer before the product fails publicly.

From the tester’s click to trained behavior

The process is always the same. One collects many questions, has both versions answer them, and randomizes the order. Then several people evaluate independently of one another. Only once hundreds or thousands of comparisons are available is the result meaningful. With fifty comparisons, a lead can be pure chance.

From the wins, one often calculates a number that follows the principle of a chess ranking: whoever frequently wins against strong opponents rises. This creates a table of many systems, even though not all of them were ever tested against each other. Well-known public rankings for language models work exactly this way.

The collected judgments are also themselves valuable training material. One teaches the system to prefer answers that humans have selected. This procedure is behind the politeness and clarity of today’s chatbots. But it has a weakness: testers like to choose the longer, more confident answer, even if it is factually wrong. A preference test thus measures readers' approval, not automatically the truth.

Two answers side by side in the chat window

Anyone who uses chatbots has probably already taken part. Some services occasionally show two answer variants and ask which one you find better. Others offer thumbs up and thumbs down. Both are forms of the same data collection, with which providers fine-tune their systems.

In press releases, the results appear as percentages. Sentences like “in 63 percent of cases testers preferred our new model” come from preference tests. Such figures deserve a critical eye: who did the testing, how many comparisons were there, and was the comparison model up to date? Without this information, the number is hardly assessable.

Outside of AI, the method is old. Food companies, publishers, and website operators test recipes, cover images, and buttons in the same way. The related term A/B test usually refers to real users who unknowingly see different versions. In a preference test, by contrast, the same person sees both variants and decides consciously.