SWE-bench Lite
SWE-bench Lite is a standardized test in which an AI system is supposed to independently fix real bugs in real software. It consists of 300 selected tasks from public coding projects and is considered the lighter, faster version of the larger SWE-bench test.
When programs have bugs, users report them in a kind of public complaint box for the project. There it might say, for example: This function returns the wrong result. A human reads this, finds the relevant spot in the program code, and changes it. SWE-bench Lite checks whether an artificial intelligence can accomplish exactly that. The system receives the bug report and the entire program code of the project, nothing else. Whether the solution is correct is then decided not by a human but by an automatic check: the project’s existing test programs must run successfully again.
Why real bugs reveal more than puzzle tasks
For a long time, AI programming abilities were measured using small, invented tasks. A typical one was: Write a function that sorts a list. Such tasks fit into a few lines and have no connection to the rest of a project. Modern models solve almost all of them, which is why the results no longer say much.
Real software projects are something completely different. They consist of thousands of files that depend on one another. The hardest part of the work is often not writing the change but finding the right spot. This is exactly the kind of search SWE-bench Lite demands. That is why the test is considered much closer to what developers actually do.
For companies, the score is also a selling point. When a provider introduces a new model, it often cites the percentage achieved on this test. Still, such figures should be read with caution. A model may have already seen parts of the tasks during training, since the projects are publicly accessible.
Cut down from 2294 tasks to 300
The original SWE-bench test contains 2294 tasks from twelve well-known Python projects. Python is a widely used programming language. Each task comes from a real, historically documented bug fix. So it is known exactly how humans solved it at the time.
Running through all 2294 tasks is expensive and takes a long time. Each one requires many requests to the model. That is why the developers selected 300 tasks that are comparatively manageable. What was sorted out were mainly cases with unclear descriptions or very extensive changes. What remains are tasks where usually only a single file needs to be adjusted.
The procedure is the same for both variants. The system receives the state of the project before the bug fix and the text of the report. It returns a change to the program code, known in technical jargon as a patch. This patch is applied, and then the tests run. Only if all tests pass does the task count as solved. There is no partial credit.
Competition for the leaderboard
The results are listed on a public leaderboard on the internet. When the test was released in 2023, the best systems solved only a few percent of the tasks. Within a few years, the scores rose to over 50 percent. This jump is often cited as evidence of how quickly AI has improved at programming.
The term is encountered mainly in announcements of models and of so-called coding agents. These are programs that combine a language model with tools, for example for searching through files. The figure also regularly appears in reports about companies such as OpenAI, Anthropic, or Google.
A common misconception is that a high score means AI can replace programmers. The test only measures narrowly defined bug fixes with clear descriptions and ready-made test programs. Designing new features, talking to customers, or planning a system does not appear in it. Because of this, extended versions now exist, such as SWE-bench Verified with human-vetted tasks.