PaperBench

PaperBench is a test task for computer programs that can write text and solve tasks. The company OpenAI released it in 2025. The program is presented with a scientific publication from AI research, meaning an academic paper describing an experiment. It must then rebuild this experiment from scratch: writing the program code, running it, and ultimately arriving at the same figures as the researchers. In total, the test comprises 20 such papers. The program is not allowed to use the authors' finished code.

Why rebuilding is the harder test

Most tests for AI systems consist of questions with a clear model solution. Math problems, programming puzzles, multiple-choice exams: you compare the answer to the correct solution, and that’s it. Such tests are becoming less and less meaningful because modern systems pass them almost flawlessly. Moreover, many of these tasks are available somewhere on the internet and thus possibly in the models' training data.

PaperBench measures something different: the ability to work on a large project over the course of hours. Rebuilding a research experiment means making hundreds of decisions. Which libraries should I use? How should I structure the data? Why does my program crash after twenty minutes? This is precisely where AI systems tend to fail more often than on individual tricky questions.

Behind this lies a very concrete question: Can AI one day conduct research independently? If a system reliably reproduces other people’s experiments, the step toward conducting its own experiments is no longer far away. That is why PaperBench is also considered an early-warning system. OpenAI itself classifies the test as part of its own safety evaluation, which the company uses to monitor how quickly its models are becoming more capable.

Evaluation based on thousands of individual points

A rebuilt experiment cannot simply be evaluated as right or wrong. That is why the authors of the original papers created a detailed checklist for each paper. These lists are structured like a tree: one major requirement breaks down into sub-requirements, which in turn break down into smaller ones. Across all 20 papers, this adds up to more than 8,000 individually checkable points.

Evaluation takes place on three levels. Does the necessary code even exist? Does it run without errors? And does the result match the paper? In the end, this yields a score between zero and one hundred percent. A human would need many hours to evaluate a single submission.

To keep this practical, a second AI model takes over the grading. This principle is called LLM-as-a-Judge, meaning a language model acting as an evaluator. Whether this automatic evaluator itself judges reliably was checked by the developers against human control assessments. The results of the first round were sobering: the best system tested achieved around 21 percent, while experienced PhD students achieved about 41 percent on the papers they worked on.

PaperBench in the debate about AI progress

Test results regularly appear in news about new AI models. Usually these concern well-known tests in mathematics or programming. PaperBench belongs to the newer generation of tests that pose longer and more open-ended tasks. When a provider advertises that its model has improved here, this is an indication of endurance on complex projects.

For investors and companies, this is more than just a number. A system that can independently carry out a research project would also take on quite different tasks in laboratories, banks, or software companies than a chatbot would. Conversely, low scores show how large the gap to human experts still is.

A common misconception is that PaperBench measures creativity or scientific originality. It does not. The test checks the clean replication of someone else’s work, not the discovery of new ideas. The test’s program code is freely available, so that other research groups can use it to test their own systems.