Ensemble

An ensemble is a group of programs that are supposed to answer the same question. Each member of the group gives its own answer. Afterwards, these answers are combined into a single result, for example through a vote or by averaging. The reason is simple: each individual program makes mistakes, but it makes different mistakes than the others. When you combine the answers, many of these mistakes cancel each other out. The principle is the same as with a jury, which judges more reliably than a single judge.

Why groups guess more reliably than individuals

Ensembles are a standard tool because they almost always bring some improvement in accuracy. In data analysis competitions, ensembles have been winning predominantly for years. The gain is rarely spectacular, but very reliable. Often it amounts to one to three percentage points more accuracy. In areas such as credit checking or medical image analysis, that can make a big difference.

A second advantage is stability. A single program sometimes reacts strongly to randomness in the data it learned from. It can perform significantly worse on slightly different examples. A group fluctuates less, because not all members are wrong in the same direction at the same time.

The price for this is computational effort. Ten programs need roughly ten times as much computing time as one. That is why ensembles are popular where accuracy matters more than speed. For applications that must respond within milliseconds, they are often dispensed with.

Voting, averaging, correcting afterwards

For an ensemble to work, the members must differ from one another. If they were all identical, voting would achieve nothing. Diversity is created, for example, by having each program learn from a different random selection of the training examples. This procedure is called bagging. A well-known implementation is the Random Forest, an ensemble of many decision trees, i.e. of programs that arrive at a verdict through yes-no questions.

Another approach is called boosting. Here the members are not built simultaneously, but one after another. Each new member focuses on those examples where the previous members were wrong. This creates a chain of corrections. Methods such as XGBoost work this way and are considered very strong for tabular data.

The way answers are combined can vary. For yes-no questions, votes are usually counted. For numerical estimates, an average is formed. Sometimes an additional small program is even trained that learns which member to trust more in which situation. A common misconception, by the way, is that an ensemble of weak members automatically becomes strong. This only holds if the errors of the members are genuinely different.

Ensembles in search engines, weather forecasts, and chatbots

Weather forecasts are the best-known example. Meteorologists run the same model multiple times with slightly altered starting values. The probability of rain shown in the weather app arises from the spread of the results. If eighty out of a hundred runs predict rain, it shows eighty percent.

Banks and insurance companies also use ensembles, for example to detect suspicious card payments. Search engines and recommendation systems combine several scoring methods into a single ranking. You rarely read the term in the news because it disappears behind product names.

With language models there is a related trick. The same model is made to solve a task multiple times, and the most frequent answer is taken. Experts call this majority voting across multiple solution paths. When a provider advertises that its system thinks longer on difficult tasks, this is often exactly the idea behind it.