MATH-500

MATH-500 is a test dataset consisting of 500 math problems used to measure how well an AI model can compute and reason logically. The problems come from a larger dataset called MATH, which was originally compiled by researchers at UC Berkeley. MATH-500 is the more commonly used, smaller selection from it — large enough for meaningful results, small enough to evaluate quickly. The problems cover areas such as algebra, geometry, probability, and combinatorics, and are divided into five difficulty levels. Many problems are on par with the level of math olympiads.

MATH-500 as a benchmark for language models

Early language models — that is, AI systems that work with text — performed very poorly on MATH-500. GPT-3 correctly solved less than 10 percent of the problems in 2021. That was a clear signal: mastering language and mastering mathematics are two different things.

Since then, MATH-500 has served as a gauge of how much progress has truly been made. When a company unveils a new model, the MATH-500 score is almost always among the first figures published. A high score shows that the model isn’t merely recognizing patterns, but can actually solve multi-step problems — step by step, like a student writing out their solution process.

The benchmark is deliberately kept difficult. It’s meant to be able to distinguish between very good models — avoiding so-called ceiling effects, meaning preventing all models from reaching nearly 100 percent and becoming impossible to compare. This works less well today than it used to: top models from 2025 are already achieving over 90 percent.

How a model is tested on MATH-500

Each of the 500 problems is presented to the model as text. The model writes a response, typically including its solution process. In the end, only the final answer is evaluated: does the number or expression match the stored model solution? The solution process itself doesn’t count — this is a well-known point of criticism, since a model could arrive at the correct result by chance even though its calculation path was wrong.

To make results more robust, many labs have the same model solve each problem multiple times — for example, 64 times — and take the most frequent answer. This method is called majority voting. It noticeably improves the score, because random errors average out. However, this makes comparisons between labs more difficult: a score of 90 percent with 64 attempts is not the same as 90 percent with a single attempt.

Another detail: the 500 problems were specifically selected in 2023 by OpenAI researcher Lightman and colleagues to form as representative a sample as possible from the original pool. Not every one of the original MATH problems is included — those for which automatic evaluation is difficult were excluded.

MATH-500 in product announcements and research reports

Anyone following tech news encounters MATH-500 mainly in press releases. OpenAI, Google DeepMind, Anthropic, and Chinese labs such as DeepSeek publish comparison tables — so-called benchmark tables — with every new model, in which MATH-500 appears alongside other tests. The term “benchmark” simply refers to a standardized test used to compare different models under the same conditions.

The model DeepSeek-R1 caused a stir in early 2025, for instance: it achieved scores on MATH-500 similar to OpenAI’s then-best model o1 — at a fraction of the training costs. Such comparisons are only publicly verifiable because benchmarks like MATH-500 exist as a shared standard.

However, MATH-500 scores should not be read uncritically. Critics point out that models may have been trained on similar problems — this is called data contamination. In that case, the benchmark doesn’t test genuine understanding but rather memory. That’s why research is simultaneously working on more difficult successor tests, such as FrontierMath, which is barely solvable even for today’s top models.