Terminal-Bench
Terminal-Bench is a test that checks how well AI programs solve real tasks on a computer by typing commands into a text window. What's measured is only whether the end result actually works.
Many professionals don’t operate a computer with a mouse, but through a plain text window. There, one types commands, for instance to copy files, launch a program, or set up a server. This window is called a terminal. Terminal-Bench is a collection of tasks that must be solved exactly in such a window — except by an AI program instead of a human. The AI is given a task described in plain language, and is then allowed to type commands itself and read the computer’s responses. At the end, it is automatically checked whether the result is actually correct. Such a standardized test with fixed tasks and fixed rules is called a benchmark.
Why typing in the text window is the tougher test
Most well-known AI tests consist of questions with one correct answer. Math problems, multiple choice, translations. In Terminal-Bench, no answer counts — a state does: Is the server running? Is the file unpacked? Does the program compile? This cannot be faked with a nicely worded explanation.
That is exactly where the appeal lies for businesses. If an AI can independently set up a system or fix a bug, it replaces hours of work by expensive specialists. Conversely, a model that can only talk about programming is worth little in operation. Terminal-Bench attempts to make this difference measurable.
The results are therefore sober. While models often score above 90 percent on knowledge questions, here they solve only a portion of the tasks, depending on the version. This is not a flaw of the test but its point: acting is harder than answering. Such benchmarks are regularly replaced by harder versions once the models become too good at them.
Container, task, verification script
Each task runs in a sealed-off miniature computer, a so-called container. This is a self-contained environment with its own operating system that can be started and discarded within seconds. This way the AI cannot cause any damage, and all test runs start under identical conditions.
The AI receives the task description and is then allowed to work in rounds. It proposes a command, the system executes it, the output goes back to the AI. From this output, it must deduce what to do next. Error messages are the norm here, not the exception.
At the end, a verification script runs, a small test program that inspects the container. It checks whether the required file exists, whether the service responds, whether the numbers are correct. The only grades are pass or fail. There are no partial points for a good approach — which, in practice, nobody pays for either.
Where these numbers show up in the news
When a company unveils a new language model today, a table of benchmark results is almost always included. Alongside math and coding tests, a value for Terminal-Bench often appears there. It is regarded as an indicator of how useful the model is as a so-called agent — that is, as a program that independently works through tasks in multiple steps.
In practice, one encounters the same capability in tools that developers use directly in the terminal. There, one describes a problem in a sentence, the program searches the code for the cause, and suggests a fix. Whether such tools work reliably depends precisely on the abilities that Terminal-Bench tests.
When reading such numbers, caution is warranted. A high score does not mean the AI masters arbitrary system tasks. Some providers let their model make several attempts or use additional helper programs. Values are only comparable if the test version and conditions are the same.