FrontierMath

FrontierMath is a collection of math problems used to test computer programs that can answer questions. Such collections of tasks are called a benchmark: all systems are given the same problems, and one compares how many they solve correctly. What is special about FrontierMath is its difficulty. The problems come from professional mathematicians, and some of them would keep even an expert busy for days. The collection was published in late 2024 by the research group Epoch AI. It came into being because older math tests were by then almost completely solved by the programs and therefore could no longer distinguish between them.

Why old math tests lost their purpose

For years, GSM8K was the standard benchmark: word problems at elementary to middle-school level. Later came MATH, with problems from competitions for high-school students. Both collections are now solved by more than ninety percent by today’s language models. A test that almost everyone passes no longer says anything about differences. Experts call this the saturation of a benchmark.

There is a second problem on top of that. The old problems have been on the internet for years, along with their solutions. AI models learn from huge amounts of text taken from the web. It is therefore hardly possible to distinguish whether a model is actually solving the problem or simply reproducing the solution from its training. This problem is called data contamination.

FrontierMath avoids both issues. The problems are newly written and are deliberately not published. And they are so difficult that for a long time the best systems solved only a small single-digit percentage. This once again creates room at the top against which progress can be measured.

Problems that check themselves

Every FrontierMath problem has a unique answer, such as a number or a mathematical object, that a program can verify. This matters because otherwise a human would have to manually check every single solution. At the same time, the answers are meant to be unguessable. That is why they are often very large or very specific values, for which guessing is practically hopeless.

The problems cover many fields, from number theory to algebraic geometry to combinatorics. Part of them require genuine proof ideas, another part demand very elaborate calculations. Tested systems are usually allowed to write and run program code in the process. A model can thus have calculations carried out instead of doing everything in its head, similar to how a mathematician uses a computer.

To keep the collection clean, Epoch AI keeps the problems under lock and key. A small part is publicly available as an example, the rest is not. It has been critically discussed that the project was partly funded by OpenAI and that the company had access to problems. A benchmark is only as credible as the independence of its examiners, and that is exactly what the debate was about.

FrontierMath in model announcements and news

The name mainly comes up when a lab presents a new flagship model. The presentation slides then show bar charts with percentage values, and FrontierMath is one of them. Anyone reading such figures should pay attention to the conditions. How many attempts per problem were allowed, was the model permitted to run code, how long was it allowed to compute? Without this information, two percentage values cannot be compared.

For investors, the benchmark is an indicator of how quickly the reasoning abilities of AI systems are improving. Jumps from a few percent to several dozen percent within a year were interpreted as evidence of rapid progress and moved stock prices.

A common misconception is that a high score on FrontierMath means general intelligence. The test measures a very narrow ability: difficult, clearly stated math problems with an unambiguous answer. It does not predict a model’s everyday judgment, reliability, or honesty. That is why FrontierMath almost always appears in reports alongside other tests, for example on coding or scientific knowledge.