SWE-bench Verified

SWE-bench Verified is a collection of test tasks for computer programs that are meant to repair software on their own. The tasks come from real projects whose source code is publicly available on the internet. Each task consists of a bug report that a human actually wrote at the time, along with the complete source code of the project. The system being tested has to figure out which file needs to be changed, and then write the change. Afterward, automated test routines run to determine whether the bug has actually disappeared. The evaluation is therefore strict and unambiguous: pass or fail.

Why fixing bugs is a harder test than code snippets

Older programming tests posed small, self-contained tasks. For example: write a function that sorts a list. Such tasks appear thousands of times on the internet, and modern language models solve almost all of them. As a result, they say almost nothing anymore about practical usefulness.

SWE-bench Verified is closer to reality. A project can consist of thousands of files that no one can read in full. The system therefore first has to search, understand, and only then write. This is exactly where many models fail, even though they handle individual functions flawlessly.

For companies, the test is therefore a point of reference for the question: Can this AI actually take work off a development team’s hands? By now, almost every announcement of a new large model cites a score on SWE-bench Verified. The values have risen quickly. A few years ago, the best software solved only a few percent of the tasks; today, top models score around two-thirds to three-quarters.

Where the 500 verified tasks come from

The basis is the older collection SWE-bench, containing over 2,000 tasks. It was compiled automatically from projects in which programmers publicly document their changes. Each task includes the original bug description and the test routines that were written after the actual fix.

This automatic selection had weaknesses. Some task descriptions were so terse that even experienced humans could not guess what was wanted. In other cases, the test routines required exactly the wording of the original author, even though other solutions would have been just as correct. A model could therefore fail without having made any mistake.

For this reason, experienced developers reviewed every task by hand. What remained were 500 tasks that are clearly described and can be tested fairly. This is exactly what the word Verified in the name means: not harder, but cleaner. Scores on the verified version are therefore higher than on the old collection and must not be compared directly.

Where the number shows up and what it conceals

SWE-bench Verified is most commonly encountered in press releases and model cards of major AI providers. Programming tools that work directly within the development environment also advertise these scores. In financial news, the number often serves as evidence that a provider is technologically ahead.

Caution is nevertheless warranted. What is tested is never a model alone, but always together with the scaffolding that shows it files and lets it execute commands. A better scaffold raises the score without the model having become any smarter. Scores are therefore only comparable under similar conditions.

A second objection concerns the origin of the data. All tasks come from public source code that the models may have seen during training. Experts refer to this as data contamination. Furthermore, the tasks cover only one programming language and a handful of projects. A high score therefore does not prove that a system has mastered every programming task.