Bayesian Prioritization

Anyone with many tasks and little time has to decide what comes first. Bayesian prioritization is a method for not fixing this order once and for all, but for constantly revising it. One starts with an estimate of how valuable or promising each option is. This estimate is based on past experience. As soon as new observations come in, they are combined with the old estimate to produce a better one. The ranking is then re-sorted. The method is named after Thomas Bayes, an English clergyman of the 18th century, whose rule describes precisely this combination of prior knowledge and new data.

Why rigid rankings usually fail

The usual alternative is a fixed list. A team gets together at the start of a quarter, sorts all projects by importance, and works through the list. The problem: the sorting is based on the knowledge available on day one. Whatever is learned in week three no longer changes the list. As a result, resources keep flowing into projects that have long since proven weak.

Bayesian prioritization turns the ranking into something dynamic. Each option carries not just an estimated value but also a measure of uncertainty. So an option can look mediocre yet be highly uncertain. Exactly such candidates are often worth a second look, because they could still turn out to be very good.

This leads to an important difference from naive optimization. Anyone who always pursues only the currently best option risks getting stuck with a mediocre choice. Experts call this the conflict between exploitation and exploration. Bayesian methods provide a computationally clean answer to this, rather than relying on gut feeling.

Prior knowledge, observation, updated ranking

The process consists of three steps that repeat. First, prior knowledge is formulated as a probability distribution. It states not only which value is likely, but also how wide the spread is. Second, data is collected: a test run, user clicks, the result of an experiment. Third, the Bayes formula combines both and yields a new, narrower distribution.

An example makes this tangible. An online shop tests five variants of a homepage. At the start, each variant is assumed to have a purchase rate of about three percent, with great uncertainty. After a thousand visitors, variant B shows four percent, variant D only one percent. The distributions shift accordingly, and traffic is directed more strongly toward B. Variant D receives fewer visitors but is not shut down immediately.

A common misconception is that prior knowledge is merely arbitrary. In fact, it loses weight with every observation. With a hundred data points, the data determine the outcome almost completely. Prior knowledge helps mainly at the start, when there is hardly any measurement data yet a decision still has to be made.

From A/B testing to model selection

The method is most commonly encountered in advertising and e-commerce. Systems that serve ads continuously reallocate budget between variants. The technical term for this is multi-armed bandit, named after slot machines with several levers. Such systems noticeably shorten testing periods, because weak variants receive less traffic early on.

In AI development, the same principle serves the search for good settings for a model. Every training run costs computing time and thus money. Instead of trying out all combinations, a Bayesian method proposes the next promising setting. This method is called Bayesian optimization and is built into common machine learning tools.

The idea also appears outside of technology. Pharmaceutical companies allocate patients in studies to treatment arms following this pattern. Investors sometimes describe their portfolio strategy using the same terms. In corporate announcements, one reads this as the phrase that projects are prioritized in a data-driven way and the order is continuously reviewed.