SWE-Bench Pro

SWE-Bench Pro is an exam task for computer programs that are meant to write code themselves. The exam consists of real bug reports from real software projects that used to be fixed by humans. The program is given the source code of the project and a description of the problem. It must then find the relevant files and fix the bug on its own. Whether the solution is correct is decided not by a human but by an automatic test run: if the software then runs correctly, the task counts as passed. The name alludes to software engineering, meaning the work of developing and maintaining larger programs.

What the test reveals about AI programmers

Almost every major AI company claims that its model can program. Such claims are hard to verify because anyone can cherry-pick examples. A shared test makes these claims comparable. That is exactly the purpose of SWE-Bench Pro: all models solve the same tasks under the same conditions.

The predecessor, SWE-Bench, had a problem: the tasks came from publicly accessible projects. Large AI models, however, learn from exactly such public data. So some models might already have seen the correct solution before. That is roughly like a student finding the exam online in advance. Experts call this problem data contamination.

SWE-Bench Pro tries to prevent this. Some of the tasks come from projects that are not freely accessible, and from company source code. In addition, the tasks are larger: it is often not about a single changed line but about many changes across multiple files. As a result, success rates come out significantly lower than in the old test.

From bug report to a passed test run

Each task consists of three parts. First, the project in the state it was in while the bug was still present. Second, the problem description as originally submitted by a user. Third, a collection of automated tests that demonstrate the bug.

In this process, the AI does not act as a pure chat partner. It runs as an agent, meaning a program that uses tools itself over multiple steps. It is allowed to search files, read source code, write changes, and start test runs. From the error messages produced by the test runs, it can learn and adjust its next attempt.

In the end, only one number counts: the proportion of tasks for which all tests pass. This evaluation is strict, but also somewhat coarse. A solution can pass the tests and still be poorly written. Conversely, a sensible change can fail because it solves the bug differently than originally intended.

Where the numbers show up

When a company presents a new language model, the press release usually contains a table of test results. SWE-Bench and SWE-Bench Pro rank among the most important entries there whenever programming abilities are at issue. Trade media and stock analysts also pick up on these percentage figures, since software development is a very large market.

One encounters the test indirectly in products as well. Coding assistants that independently fix bugs or submit proposed changes are tested internally against such task collections. Rising scores are an indication that these tools are becoming more reliable in everyday use.

A common misconception is worth mentioning in closing. A high score does not mean that an AI replaces the profession of a developer. The test always provides a clearly defined problem with ready-made evaluation criteria. In real working life, precisely these specifications are missing: one must first figure out what should even be built.