SciCode

SciCode is a collection of exam-style tasks for computer programs that can write text and answer questions. In the field, such collections of tasks are called benchmarks. They work like a school test: all test-takers get the same tasks, and at the end there is a score for comparison. In SciCode, however, the tasks are very specialized. The program being tested is meant to solve computational problems from science by independently writing program code to do so. The tasks come from real researchers and were derived from academic papers, including ones from physics, chemistry, biology, and mathematics.

What SciCode reveals about real research work

Many well-known coding tests consist of small tasks like the ones you’d find in a beginner’s course. One example would be: sort a list of numbers. Today’s systems solve such tasks almost flawlessly. The scores there are so high that it’s hard to tell the models apart anymore. A test on which everyone gets a top grade doesn’t say much.

SciCode deliberately goes deeper. Here, it’s not enough to know a programming language. One must also understand which formula is physically applicable in the first place, and then correctly translate it into computational steps. That is exactly what researchers do every day when they program simulations or analyses. So the test doesn’t measure textbook knowledge, but something that comes close to real lab work.

The results are therefore sobering and revealing at the same time. Even the strongest systems initially solved only a single-digit percentage of the complete main tasks. This is an important counterpoint to headlines that portray AI as a ready-made research assistant. SciCode soberly shows just how large the gap to that goal still is.

Split-up tasks and testing against computed results

Each main task in SciCode describes a larger scientific problem. It is broken down into several smaller sub-steps, usually a handful per task. In total, the collection contains about 80 main tasks and around 300 sub-steps. The system being tested works through the sub-steps one after another, building on its own earlier solutions along the way. So a mistake at the beginning carries through the rest.

What is assessed is not whether the code looks nice. Instead, it is run, and the computed numbers are compared with reference values that experts have determined beforehand. If the result deviates, the step is considered wrong. This kind of check is strict, but it cannot be tricked by nicely worded explanations.

Each task also comes with a scientific background text. The test can be carried out with or without this aid. Without the background, the model must bring the necessary expert knowledge itself. Comparing both variants shows where the problem lies: missing knowledge or coding ability.

SciCode in model announcements and leaderboards

When a company introduces a new language model, it almost always publishes a table of benchmark results. Alongside well-known names, SciCode now frequently appears there. A percentage figure next to it indicates the share of correctly solved sub-steps or main tasks. It’s worth paying close attention when reading this, since the two figures differ substantially.

For investors, the test is interesting because it targets a potential market. Companies, universities, and pharmaceutical firms hope for software that speeds up research work. Progress on SciCode is seen as a sign that this hope is becoming more realistic.

A common misconception should not be overlooked here. A high score on SciCode does not mean a system can conduct research on its own. It solves tasks whose solution paths humans already knew and had written down. Asking new questions is something different from correctly reproducing known calculations.