AIME

AIME stands for American Invitational Mathematics Examination. It is a mathematics competition in the USA to which only students who performed well in a qualifying round are invited. It consists of 15 problems, for which three hours are allotted. Every solution is an integer between 0 and 999, so there are no multiple-choice options and no essay text. It is precisely this feature that has made the competition interesting for the AI industry: companies have their programs solve the problems and report how many were solved correctly. If a report states that a new model achieves “90 percent on AIME 2025,” this is the competition being referred to.

Why a student competition of all things became a benchmark

Anyone wanting to compare the capabilities of an AI system needs tasks with unambiguous solutions. With a poem or a summary, people disagree about what counts as good. With a number between 0 and 999, there is no disagreement. A checking program compares the output number with the correct one and counts the hits. Such standardized collections of tasks are called benchmarks, i.e., comparison tests.

On top of that comes the level of difficulty. AIME problems are not a matter of plugging values into formulas; they require several steps of reasoning in sequence. A typical example: one must first recognize a geometric relationship, derive an equation from it, and then solve it. Anyone who makes a single calculation error along the way ends up with a wrong number. The test therefore does not measure knowledge, but the ability to sustain a longer chain of reasoning without error.

This is precisely where AI systems lagged behind for years. As late as 2023, leading chat programs solved only a small fraction of the problems. Since 2024, providers have been reporting figures above 80 or 90 percent. This leap is one of the reasons why so much is being written about so-called reasoning models — that is, models that write out extensive intermediate steps before giving an answer.

From problem text to checked number

The procedure is simple. The model is given the problem text, usually with an instruction to write the final solution in a specific format. The model then typically generates several paragraphs of working. From this output, the last number stated is automatically extracted and compared with the official solution. The working itself is normally not evaluated at all.

There is an important subtlety in the measurement. Because the same question does not always produce the same answer from a model, each problem is run multiple times. In the common “pass@1” method, results are averaged across these runs. Some providers instead report figures in which the model solves the problem, say, 64 times and the most frequent answer is counted. Such figures are naturally higher and are not directly comparable to the others.

Another problem is called contamination. Old AIME problems, complete with solutions, are available on the internet and thus end up in training data. A model might then recall the answer instead of deriving it. For this reason, it is mainly the most recent year’s problems — those published after the training cutoff date — that are used for testing.

Reading AIME figures in product announcements correctly

You will most often encounter AIME in the tables that AI companies publish for new models. There, the competition appears alongside other tests for coding or the natural sciences. Trade media and financial reports also pick up the figure, since it fits easily into a headline.

When reading such figures, skepticism is warranted on two points. First, it should be stated which year’s exam was measured and how many times the problems were run. Second, keep in mind that AIME consists of only 15 problems per exam. A single additional correctly solved problem already shifts the result by nearly seven percentage points. Small leads between two models therefore say little.

A common misconception is also that a high AIME score means general intelligence. The test measures closed-form calculation problems with a known solution. Whether a model reliably summarizes sources in everyday use or admits its mistakes is an entirely different matter. Because top scores are now approaching the 100 percent mark, AIME is also losing its informative value and is gradually being supplemented by harder collections of problems.