SWE-bench

SWE-bench is a collection of evaluation tasks for AI systems that write program code. Each task is a real bug report from a freely accessible software project, meaning a program whose source code is publicly viewable on the internet. The AI system receives the state of the project before the fix and the description of the bug. It must figure out on its own which files are affected and make a change there. Whether the change is correct is not decided by a human but by an automated test run: the project comes with small test programs that check the software’s behavior. If they run without errors after the change, the task counts as solved.

Why real bug reports are harder than coding exercises

Older tests for coding AI posed small, self-contained tasks: write a function that sorts a list. Such tasks fit on half a page and have nothing to do with the everyday work of software companies. There, the work usually consists of finding a single broken spot in a grown project with thousands of files. SWE-bench measures exactly this ability.

That is why the test has become an important benchmark for the industry. When a company introduces a new language model, the SWE-bench score is almost always part of the announcement. The numbers are well comparable because all providers use the same tasks. An increase from 20 to 60 percent solved tasks within a few years is solid evidence of progress.

Economically, this is significant. Investors and companies read from such figures how much programming work AI could take over. At the same time, researchers warn against confusing a single percentage with general competence. A system that fixes bugs has not thereby demonstrated that it can meaningfully design a new program.

From bug report to passed test run

The tasks were derived from the version history of well-known projects, many of them in the Python programming language. The developers searched for cases where a reported bug was actually fixed through a code change. This human solution is removed and kept as a secret reference. What remains is the project in its broken state plus the original bug description.

The AI system then usually does not operate as a pure text generator but as what is called an agent. This is a program that runs the model in a loop: read files, search, write a change, run tests, evaluate error messages, refine. At the end, it submits a patch, meaning a precise list of the changed lines. This patch is applied to the original project.

Now two groups of tests run. One group must have failed before and succeed afterward, otherwise the bug was not fixed. The other group was already fine before and must remain so, so that nothing new breaks. The task only counts if both are true. A known problem with the test: some bug reports are phrased so tersely that even humans could hardly guess the expected solution. That is why there is a vetted subset called SWE-bench Verified with 500 tasks checked by experts.

Where the percentage shows up in headlines

Most commonly, one encounters SWE-bench in product announcements. When an announcement states that a model solves “74 percent on SWE-bench Verified,” this is the test being referred to. Because different providers use different variants and agents of varying sophistication, two numbers are only somewhat comparable. It is almost always worth checking the footnote.

Indirectly, one notices the effect in tools that programmers use daily. Assistants that independently search for bugs and suggest changes within the development environment have been trained and measured on exactly such tasks. Services that automatically respond to bug reports in a project also trace back to this line of research. Anyone who codes themselves can try out the idea on a small scale: describe a bug, let the model read the code, then run the tests.