BLEU Score

When a computer program transfers a sentence from one language into another, one question immediately arises: how good is the result? Humans could judge this, but that costs time and money. The BLEU score is a computational procedure that handles this judgment automatically. You feed it the program’s output and one or more translations that humans previously produced. It compares the two and outputs a number between 0 and 1, often also given as a percentage from 0 to 100. The higher the number, the more closely the machine translation resembles the human model.

Why researchers need a number at all

Anyone who wants to improve a translation system must be able to measure whether a change actually helps. Without a metric, only the impression of individual test persons remains. The BLEU score instead delivers a result that anyone can recompute using the same data. It was precisely this repeatability that made it the standard in research from 2002 onward.

A second reason is speed. A team can test hundreds of variants of a system per day if the evaluation takes seconds instead of weeks. Human reviewers would be far too slow for that. That is why major advances in machine translation only became possible thanks to such automatic measures in the first place.

However, it is important to put this in context: BLEU does not measure whether a translation is elegant or comprehensible. It only measures agreement with a reference. A score of 40 is considered solid for many language pairs, and a score of 60 very good. Absolute numbers are nevertheless hardly comparable, because they depend heavily on the test text.

The comparison of word sequences

The procedure breaks both sentences down into short word sequences. Such a sequence is called an n-gram: a single word, a pair of words, a group of three or four. BLEU then checks how many of these sequences from the machine translation also appear in the human reference. The overall score is derived from these match rates.

Two corrections prevent cheap tricks. First, a word only counts as often as it actually appears in the reference. So a program cannot score points by outputting “the” twenty times. Second, there is a penalty for outputs that are too short. Because a system that outputs only a single safe word would otherwise achieve a perfect match rate.

This gives rise to the best-known weakness of the procedure. A good translation that uses different words than the reference receives a low score. “The car is fast” and “The vehicle moves quickly” mean the same thing, but hardly share any n-grams. BLEU does not understand meaning, it counts matches. That is why multiple human references are often used simultaneously, in order to allow for different phrasings.

BLEU in model announcements and its successors

The value regularly appears in technical papers and press releases about new language models. Companies like Google or Meta show tables in which their system achieves a BLEU score two points higher than the competition. Anyone reading such reports should know that these points only mean something within the same test dataset. A comparison between two different studies is usually worthless.

In products themselves, you rarely encounter the number. It sits in the engine room of development, not on the surface of translation apps. Nevertheless, it has shaped their quality, because developers optimized precisely toward this value for years.

Today BLEU is increasingly under criticism. Newer measures such as COMET or BERTScore do not compare words but the meaning of sentences, and correlate better with human judgments. With modern chatbots that formulate freely, pure word counting fails anyway. Nevertheless, BLEU remains in use as a fast, well-understood benchmark for comparison.