SWE-Pro

SWE-Pro is a testing procedure for programs that write or repair software on their own. Such testing procedures are called a benchmark: a fixed collection of tasks that all participants solve under the same conditions. The tasks in SWE-Pro come from real, publicly viewable software projects. A tested system receives a bug description and the complete program code of the project. It must figure out which file is faulty and change the relevant spot so that the bug disappears. Whether this succeeds is decided not by a human but by automated test programs that the project already has.

What the test says about AI programmers

Almost every vendor claims their model can program. Such claims are worthless as long as no one can verify them. A benchmark makes the claim verifiable because everyone receives the same tasks. One can then say: System A solves 43 out of 100 tasks, System B only 28.

The name hints at the difference from older tests. “Pro” stands for professional: the tasks come from projects that paid developers work on. Such code is messy, has grown over time, and is poorly documented. Earlier programming tests, by contrast, often consisted of small, cleanly formulated individual tasks. Today’s models solve these tasks almost completely, which is why they hardly differentiate between systems anymore.

For companies, the score from such tests serves as a decision-making aid. Anyone buying an AI tool for their development team wants to know whether it helps with real work. A high score on a test with exercise tasks says little about that. That is why realistic benchmarks now carry more weight in trade news than they used to.

From bug report to green test run

The process is modeled on everyday life in a development team. Users report bugs in a public registry, the so-called issue tracker. A developer reads the report, searches for the cause, and submits a fix. Precisely these real bug reports and the corresponding real fixes form the basis of the tasks.

During the test, the AI system receives only the bug report and the state of the project before the fix. The human developer’s solution remains hidden. The system must work its way through thousands of files and decide for itself where to search. In the end, it delivers a change, referred to in technical jargon as a patch.

Afterwards, the test machine runs the project’s test programs. Certain tests that previously failed must now pass. At the same time, no previously working test may break. Only if both conditions are met does the task count as solved. Partial solutions or well-intentioned approaches earn no points.

SWE-Pro in product announcements and leaderboards

You will mainly encounter the term in reports about new language models. When a vendor unveils a model, they usually show a bar chart with benchmark results. Scores from the SWE family now rank near the top there, because programming assistance is a large market. Stock market reports also pick up on such figures when they want to justify a company’s lead.

You encounter the test indirectly in tools that developers use every day. Programs that suggest code or fix bugs on their own are tuned against such benchmarks. If a model improves on the test, this is often noticeable in practice. However, this is only a correlation, not a guarantee.

A common misconception is to read the percentage as a measure of human replacement. The test measures narrowly defined repair tasks with a clear success criterion. Advising a team, designing an architecture, or clarifying a vague requirement do not appear in it. A second problem is called contamination: if the tasks were already present in the model’s training material, it may know the solution by heart. That is why newer benchmarks deliberately include tasks that are more recent than the training data.