Real-SWE

Software is written by people in teams, and in the process, bugs and open feature requests constantly pile up. These are recorded in a kind of public task list, often on platforms like GitHub. Real-SWE is the name given to testing procedures in which an AI system is supposed to work on exactly such real, already documented tasks from real projects. The name comes from the English “Real Software Engineering,” that is, actual software development. The opposite of this are small, specially invented practice exercises, of the kind familiar from computer science classes. The demand, then, is: not to solve a puzzle, but to make a real change in a large, organically grown program that actually works.

Why practice exercises overestimate capabilities

For years, the programming abilities of AI systems were measured using short, clearly defined tasks. A typical example was: write a function that sorts a list. Such tasks appear millions of times on the internet, and the models have seen them during training. High scores therefore said little about whether a system is useful in a real project.

Real projects are built differently. A medium-sized open-source program quickly has a hundred thousand lines of code, spread across thousands of files. Anyone who wants to fix a bug there must first figure out where the problem is actually located. Often the bug description is vague, something like “program crashes on large files.” This search is frequently harder than the actual fix, which in the end might only span three lines.

For companies, a lot of money hinges on this question. If an AI system takes over a noticeable share of everyday maintenance work, that changes the cost calculations of entire development departments. That is why Real-SWE results regularly show up in quarterly reports and product announcements. And that is why it is worth taking a close look at what was actually measured there.

How a real task becomes an exam question

The tasks come from the history of real projects. One looks for a reported bug for which a human later wrote a fix. The fix is removed, the bug report remains. The AI system is now given the state of the project from back then along with the description of the problem. It is supposed to figure out on its own which files it needs to change.

Evaluation is automatic. Software projects contain what are called tests: small auxiliary programs that check whether everything still works as expected. A test for the specific bug report must pass after the change. At the same time, the existing tests must not break. So only the outcome counts, not whether the solution approach resembles the human one.

A well-known pitfall here is contamination of the test data. If the original solution has long been publicly available on the internet, the model may have read it during training. In that case, one is testing memory rather than ability. Newer variants therefore use only tasks that arose after the model’s training cutoff. As a result, the scores are often noticeably lower.

Real-SWE in product announcements and the news

When a provider introduces a new language model, the announcement almost always includes a percentage figure for solved real programming tasks. Within just a few years, these numbers have risen from single digits to over seventy percent. They are read by trade media and financial markets as a gauge of progress. Nevertheless, they should be interpreted with caution.

This is because the values depend heavily on how much help the system is given. Some measurements allow multiple attempts and select the best one. Others give the model tools with which it can search the code and run tests itself. A direct comparison of two figures from different announcements is therefore often misleading.

In everyday life, you encounter the same idea in programming assistants built into development environments. Such tools are assigned a bug ticket and independently propose a change. A human reviews the proposal and either adopts it or not. Real-SWE is thus less a single product than the yardstick against which these tools are measured.