SWE Marathon

Programs are written in text files, known as code. Anyone improving software changes this text in many places at once. SWE Marathon is a test for AI systems that are meant to take on exactly that task. The name alludes to the distance: it’s not about a quick little fix, but about assignments that would keep a human occupied for several days. The AI is given a real, large software project along with a task, such as fixing a bug or building in a new feature. In the end, all that counts is whether the software actually works afterward. Such comparative tests are called benchmarks, meaning standardized sets of tasks against which different systems can be measured.

Why endurance is harder than cleverness

Older programming benchmarks consisted of short snippets. A typical task was: write a function that sorts a list. Today’s models pass such tests almost completely. But they say hardly anything about whether an AI can take over the work of a developer.

The real effort in software development lies elsewhere. You have to understand a project with hundreds of thousands of lines of code, find the right spot, and avoid side effects. This is exactly where AI systems often fail. They start out strong, but after many steps they lose sight of the goal or fix things that nobody broke.

For companies, this endurance is the decisive point. An AI that works reliably for ten minutes needs constant supervision. An AI that holds up for eight hours fundamentally changes the division of labor within a development team. That’s why a test like SWE Marathon is considered a meaningful indicator of how close the technology really is.

How a marathon task proceeds

The AI doesn’t work in a chat window, but in a self-contained work environment. There it has tools: it can read files, change text, run commands, and start test programs. These test programs automatically check whether the software still does what it’s supposed to. A system that acts independently like this is called an agent.

The process is a long loop. The AI thinks, makes a change, runs the tests, and reads the result. From the error messages it derives the next step. This repeats hundreds or thousands of times, without any human intervening.

In the end, only the outcome is evaluated. If all the intended tests pass, the task is considered solved; otherwise, it is not. This strictness is intentional: a convincing-sounding explanation earns no points if the code doesn’t run. One common misconception, by the way, is that computing speed is what matters most here. What’s more important is that the model retains context over long stretches, meaning the memory of everything it has already done.

What news about AI programmers is really talking about

When providers like OpenAI, Anthropic, or Google introduce a new model, they almost always cite figures from programming benchmarks. SWE Marathon belongs to a newer generation of such tests that focus on long, realistic assignments. Rising percentage figures in these tables are the reason why some companies are publicly questioning their hiring plans for junior developers.

For investors and observers, these numbers serve as a fever thermometer for the industry. But they should be read with caution. A model can be optimized for a specific test set and still disappoint in practice. Moreover, a percentage figure reveals nothing about cost: a marathon run consumes a great deal of computing time, and thus money.

In everyday life, you encounter the same technology in a scaled-down form. Programming assistants in development tools already independently edit multiple files and run tests today. What is tested under laboratory conditions in SWE Marathon usually ends up in such products one to two years later.