Consensus Output

Programs that generate texts or answers don’t always work the same way. If you ask them the same question twice, you often get two slightly different answers. This is because these programs roll the dice between several possibilities for each word. A consensus output exploits exactly this. The same task is solved five, ten, or twenty times, and one looks at which result appears most often. This majority result then counts as the final answer.

Why majorities are more reliable than single answers

Errors in such programs usually arise randomly and at different points. If one computational path takes a wrong turn at some point, the next run often doesn’t. Wrong answers therefore spread out broadly, while correct answers accumulate at one point. For a computational task there is exactly one correct result, but dozens of ways to solve it incorrectly.

This is the same effect as with a class test that is graded through group work. One person alone can easily overlook a mistake. Ten people calculating independently rarely all arrive at the same wrong value. Studies on math problems show improvements of several percentage points when the majority from forty runs is taken instead of a single answer.

An important distinction: a consensus output does not make the program smarter. It cannot find anything it couldn’t already find on its own. If the program fundamentally misunderstands a matter, all twenty runs will be uniformly wrong. The majority is then confident and still off the mark. This is the most common misconception about this method.

From dice-rolling to tallying

For the method to work at all, the runs must differ from one another. This is ensured by a control that determines how much the program deviates from the most probable path when generating words. If this control is set to zero, the same result comes out every time, and the voting would be pointless. A medium value creates enough diversity without the answers becoming incoherent.

Afterward, the results are compared and tallied. For numbers or yes-no questions, this is simple: one counts how often each value occurs. For running text, it is more difficult, because two answers can mean the same thing but be phrased differently. In that case, either a second program checks for agreement in content, or only the decisive concluding statement is compared.

The price for this is computing time. Twenty runs cost roughly twenty times as much as a single one. That’s why the method is used selectively, not everywhere. Some systems stop early as soon as the first five runs are unanimous, and only keep computing in case of disagreement.

Consensus outputs in products and reports

In reports about new AI models, the term often appears in the fine print of test results. If it says something like “maj@32,” the task was solved 32 times and the majority was scored. Such values are not directly comparable to results from a single run. Anyone comparing models should therefore pay attention to this detail.

The principle is also built into products without being visible. Some chat systems compute several solution paths in the background for difficult questions and only display the majority result. Something similar applies to automatic translation or speech recognition. Evaluations of job applications or invoices at companies are also sometimes run multiple times to avoid slip-ups.

Related but not identical is the term ensemble. There, several different models are made to vote, not the same model multiple times. Both approaches pursue the same goal: random slip-ups should cancel each other out. Systematic errors shared by all participants remain in both cases.