Coding Benchmark

A coding benchmark is a fixed collection of programming tasks used to compare computer programs that themselves write program code. All systems being tested receive exactly the same tasks, so that the results are comparable. A task usually consists of a short description, for example: write a function that finds the second-largest number from a list of numbers. The written code is then automatically executed and checked against ready-made verification steps, so-called tests. If the code runs correctly on all tests, the task counts as solved. At the end there is a number, for example: 62 percent of tasks solved.

Why companies advertise with percentages from code tests

Programming is one of the few skills where right and wrong can be determined almost automatically. With an essay, people argue about the grading. With code, the computer decides: it runs or it doesn’t. That’s why coding benchmarks are especially popular among AI system providers when they present new versions.

There is also an economic reason. Developing software is expensive, and programmers are well paid. If an AI system takes over a measurable part of this work, that can immediately be translated into money for companies. This is exactly why the stock prices of tech companies react more strongly to such test results than to many other announcements.

However, one should not overestimate these numbers. A high score does not mean that the system can independently develop an entire program. What is tested are usually small, clearly defined tasks. The actual work in a project often consists of figuring out what should even be built in the first place.

From task description to percentage

The process is nearly the same for almost all coding benchmarks. First, there is a collection of tasks with matching tests, compiled by humans. The AI system receives a task as text and returns code in response. This code is executed in an isolated environment so that it cannot break anything. If it passes all tests, the task counts as a point.

The tests are the actual core of it. They don’t check whether the code looks nice, but whether it delivers the expected results. For the second-largest task, one would check, for example: What happens with a list containing only one number? What about duplicate values? It is precisely on such edge cases that AI systems often fail, even though the code looks correct at first glance.

A well-known problem is called contamination: the tasks are often freely available on the internet. AI systems learn from enormous amounts of text from the web and may have already seen the solutions. In that case, one is not testing ability but memory. It’s like a class test whose answers the student has already memorized beforehand. That’s why new benchmarks with fresh tasks are constantly being created, for example from current bug reports of real software projects.

Well-known test collections and where their scores appear

Two names appear especially often in the news. HumanEval consists of around 160 small programming tasks and is the older, simpler test. SWE-bench works with real bug reports from large open-source projects, meaning freely accessible software. There, the system has to find its way through thousands of files and make a change in the right place. That’s why the percentage scores there are noticeably lower.

You will encounter these numbers mainly in press releases, stock market announcements, and comparison tables from providers like OpenAI, Anthropic, or Google. Rankings on the web, so-called leaderboards, also sort models by such results. Anyone reading marketing claims like “new best score in programming” should always check which benchmark is meant.

In practice, this is noticeable in programming aids like GitHub Copilot or in chat assistants that suggest code. However, a benchmark can only say so much about whether such a tool is genuinely helpful in everyday use. Often more useful is the question of how reliably it works on one’s own tasks and how easily errors can be found.