MMLU-Pro

MMLU-Pro is a collection of exam questions for computer programs that can understand and write text. Such programs are called language models; well-known examples are ChatGPT or Claude. The test contains more than 12,000 questions from subjects such as mathematics, physics, law, medicine, economics, and history. Each question has ten possible answers, of which exactly one is correct. The program is made to answer all the questions, and the hits are counted at the end. The result is a percentage value that can be used to directly compare different programs.

Why the old MMLU test had run its course

MMLU-Pro is the successor to a test called MMLU from 2020. It consisted of similar questions, but had only four answer options per question. Back then, this was a genuine hurdle: the best programs barely scored above what pure guessing would achieve. Four years later, many models reached scores around 88 percent, and the gaps between them shrank to one or two percentage points.

If all the students in a class have nearly full marks, the grade no longer says anything about the differences between them. That was exactly the problem with MMLU. On top of that, some of the questions were flawed or allowed for multiple defensible answers. And because the questions were freely available on the internet, they ended up in the models' training material. A model could then simply know the answer by heart.

MMLU-Pro was developed in 2024 at the University of Waterloo in order to reintroduce a gap between the models. It worked: when switching from MMLU to MMLU-Pro, scores typically dropped by 15 to 30 percentage points. For companies, research, and the press, the test is therefore one of the standard figures cited whenever a new model is introduced.

Ten answers instead of four

The most important change sounds trivial, but has a strong effect. Instead of four, there are now ten answer options. Anyone guessing blindly now only hits the mark in 10 percent of cases instead of 25 percent. Luck is largely ruled out as an explanation for a good score.

Second, easy knowledge questions were weeded out and replaced with tasks that require several calculation or reasoning steps. A large portion comes from university exams and textbooks. Human reviewers also sorted out or corrected thousands of flawed questions from the old dataset.

This leads to an interesting effect: models perform noticeably better on MMLU-Pro when they are allowed to write out their solution path beforehand. This step-by-step thinking is called chain-of-thought. With the old MMLU, this barely helped, because it mostly required pure recall of knowledge. MMLU-Pro therefore measures not only what a model knows, but also how well it calculates and reasons.

MMLU-Pro in model announcements and leaderboards

The term mainly comes up when a company introduces a new language model. In the tables of press releases, MMLU-Pro usually appears in the first or second row, alongside tests for programming and mathematics. Trade articles and financial news also cite the figure to argue who is currently in the lead. Public leaderboards, such as those on the Hugging Face platform, list it as well.

When reading such figures, some skepticism is warranted. A difference of one percentage point means little, since the result also depends on how the questions are phrased. What matters more is whether a model is five or ten points ahead of another. And a high score does not guarantee that the model feels pleasant to use in everyday life or rarely makes things up.

Another common misconception is that MMLU-Pro is a kind of intelligence test. It only tests multiple-choice questions from academic subjects. It says nothing about creativity, conversational skill, or reliability in real work tasks. And this test, too, will age: once the top models are close to 90 percent, the next, harder successor will be needed.