Bernoulli Distribution

Some processes have only two possible outcomes. A coin shows heads or tails. An email is spam or it isn’t. A customer clicks on an ad or they don’t. For such cases, mathematics offers a very simple computational model, named after the Swiss mathematician Jakob Bernoulli. One fixes a number p between 0 and 1: the probability of the one outcome, usually called the hit. Everything else follows from that, because the second outcome automatically has probability 1 minus p. This makes the Bernoulli distribution the smallest possible description of randomness that still says anything at all.

The building block behind every yes-no statistic

The Bernoulli distribution is boring on its own. It becomes interesting because larger models are built from it. If you repeat the same trial a hundred times and count the hits, you arrive at the binomial distribution. This tells you, for example, how likely it is to see heads exactly 58 times out of a hundred coin tosses. Almost every statistic about yes-no questions has the Bernoulli distribution as its basic building block.

It also constantly appears in artificial intelligence. Many models are meant to make a decision: is there a tumor in the image or not? Is this transfer fraud or not? The model does not output a hard answer but a number like 0.87. That very number is the p of a Bernoulli distribution. So the model is not claiming certainty, but rather stating a probability of a hit.

This perspective has a practical benefit. Because the result is a probability, one can freely choose the threshold. In a medical pre-screening, an alarm might already be triggered from 0.2 onward, because a missed case is worse than a false alarm. For a spam filter, one chooses a high threshold so that no important email disappears.

A single parameter and what one calculates from it

Formally, a Bernoulli-distributed quantity takes only the values 1 and 0. The 1 represents a hit and occurs with probability p. The 0 represents a miss and occurs with probability 1 minus p. The model needs no more ingredients than that. This is why it is called a distribution with only one parameter.

The expected value, i.e. the long-run average, is simply p. If you flip a fair coin very many times and record a 1 for heads, the mean comes out to about 0.5. The variance, a measure of fluctuation, is p times (1 minus p). It is greatest at p equal to 0.5 and approaches zero as p gets close to 0 or 1. This matches intuition: an event that almost always occurs is hardly surprising.

When training AI models, this distribution becomes the evaluation rule. The model predicts a probability, and then the actual outcome is observed. If the predicted probability for the outcome that occurred was high, there are few penalty points. If it was low, there are many. This measure is called binary cross-entropy and stems directly from the Bernoulli distribution. It’s important to keep the distinction clear: the distribution describes a single trial, not a series. Anyone counting hits across many trials has already moved to the binomial distribution.

From A/B tests to click models

In everyday life, this model shows up wherever something is counted that knows only two states. Online shops test two variants of a page against each other, the so-called A/B test. Each visitor either buys or doesn’t, so that’s one Bernoulli trial per person. Only the sum across thousands of visitors reveals which variant performs better.

In business news, the same calculations lie behind terms like probability of default. A borrower repays or doesn’t. Banks estimate a p for each customer and use it to calculate how much money they need to set aside as a buffer. Insurers work the same way: a claim occurs or it doesn’t.

A common misconception is assuming that the individual trials are always independent of one another. That only holds if one outcome does not influence the next. In a financial crisis, many loans default at the same time because the same underlying cause is at work. If one still calculates using independent Bernoulli trials in such a case, one massively underestimates the risk. That was precisely one of the mistakes that became visible in 2008.