Terminal-Bench-Science

Terminal-Bench-Science is a collection of test tasks for AI programs. What is being tested is not knowledge, but action. The AI works in a terminal, that is, a text window in which commands are typed to the computer as lines. There it is meant to solve tasks from everyday research work: evaluating measurement data, getting a program to run, carrying out a calculation. At the end, an automatic test checks whether the result is correct. Such standardized tests are called benchmarks, and this one is tailored to the natural sciences.

Testing knowledge is no longer enough

For a long time, AI models were evaluated using knowledge questions. They were given physics problems or chemistry questions with four answer options, and the hits were counted. For the best models, these scores are now close to full marks. This makes it hardly possible anymore to distinguish which model is truly better. The test is saturated, as experts put it.

At the same time, what AI is used for has shifted. Companies don’t want models that answer questions, but so-called agents. An agent is an AI program that independently works through a task in multiple steps. It is allowed to use tools, open files, launch programs, and correct its own mistakes. Whether a model can do this is something a multiple-choice question reveals nothing about.

It is precisely this gap that Terminal-Bench-Science is meant to close. A task counts as solved only if a correct file or number comes out at the end. Roughly correct explanations don’t count. This makes the results hard-edged and difficult to gloss over. For investors and editorial teams this is interesting, because such figures show whether expensive models really deliver in everyday work.

How a test task proceeds

Each task starts in an isolated working environment, usually a container. This is a small computer made of software, separated from the rest of the machine. It already contains files, data sets, and the necessary programs. The AI receives a task description in plain language. After that, it is on its own.

Now a loop of command and feedback begins. The AI types a command, the terminal responds with text, often with an error message. From this response it derives the next step. This can take dozens of rounds. This ability to draw the right conclusion from an error message is the actual core of the test.

At the end, a test script runs that a human wrote beforehand. It checks, for example, whether the generated result file contains the expected values. There is only pass or fail. The final score is the proportion of solved tasks, often given as a percentage. Important for distinction: Terminal-Bench-Science measures scientific workflows, while the better-known test SWE-Bench focuses on bugs in software projects.

The test in product announcements

Most often, this name is encountered in announcements about new models. When a provider introduces a new system, it usually shows a table with benchmark scores. Between math and coding tests, the category of scientific terminal tasks increasingly appears there as well. A jump from, say, 20 to 40 percent is considered a strong argument.

Such figures should be read with caution. Models are specifically improved toward known tests, which devalues a benchmark's worth over time. Moreover, many tables don’t disclose how many attempts per task were allowed and how much compute time it cost. A model that computes twice as long has an unfair advantage. Serious comparisons state these conditions alongside the results.

The topic becomes practically relevant in labs and research departments. There is a lot of routine computer work there: reformatting data, repeating analyses, installing software. If AI agents reliably take over these steps, it saves real working time. Tests like this one are an attempt to honestly measure how far the systems still are from that.