A/B Test

A company wants to know whether a green or a blue buy button works better. Instead of debating it, they randomly split their visitors into two groups. Group A sees the green button, Group B the blue one. Afterward, they compare in which group more people actually made a purchase. This exact procedure is called an A/B test. At its core, it’s the same principle as a drug trial: two groups, a single difference, a measured outcome.

Why opinions are poor advisors

Humans are very bad at estimating what other people will click on. Experienced designers regularly get such predictions wrong. Major internet companies report that only a small fraction of their ideas actually improve the numbers. Many changes have no effect at all, and some are even harmful. Without measurement, no one would notice the difference.

The A/B test replaces a ranking based on hierarchy with a ranking based on data. It’s not the boss who decides which text goes on the button, but the behavior of the users. This makes product development slower, but more reliable. Companies like Amazon, Netflix, or Booking therefore run thousands of such tests per year.

This is especially important for AI products. Whether a new model version really responds more helpfully can only be checked to a limited extent in the lab. Only comparison with real users shows whether people reuse the answers more often or abandon the conversation sooner.

Randomness, target metric, and dealing with fluctuations

The random split is decisive. Only if both groups are, on average, composed identically can the difference in outcome be attributed to the change. If Group A were formed in the morning and Group B in the evening, the test would be worthless. Then one would have compared times of day, not button colors.

Beforehand, a target metric is defined, i.e., the number by which success is measured. This could be the purchase rate, the time spent, or the number of drop-offs. Without a fixed target metric, one keeps searching afterward until some number looks good. This exact mistake is one of the most common in practice.

Moreover, measured values always fluctuate a little, even without a real difference. That’s why statistics are used to calculate how likely it is that the measured result is pure chance. Small groups yield unclear results. To reliably detect a difference of one percent, tens of thousands of users per group are often needed. Anyone who stops the test early because one variant is currently ahead often deceives themselves.

From the app interface to the chatbot model

Almost every major app is constantly testing on its users. Different homepages, different orders in the feed, different prices for the same trip: much of this runs as an A/B test. You usually don’t notice it because you only see your own variant. If a friend describes an app as looking different, sometimes exactly this is behind it.

In the AI industry, the term appears in two contexts. First, when rolling out new model versions: a small portion of requests go to the new model, the rest to the old one. Second, on comparison platforms where users see two anonymous answers and choose the better one. Such duels give rise to the well-known rankings of language models.

However, the A/B test has limits. It shows which version performs better, not why. And it usually measures only short-term reactions. A sensational headline can increase click numbers today and reduce trust over months. That’s why companies combine tests with surveys and long-term metrics.