HumanEval

HumanEval is a collection of 164 small programming tasks. It serves as a test for computer programs that can write program code themselves. Each task consists of the beginning of a program and a short description in plain English, for example: “Return the smallest number from a list.” The AI system is meant to complete the rest of the solution. The solution is then actually executed on a computer and checked against prepared test cases. Only those who pass all the tests get the point — being half right doesn’t count. The collection was published in 2021 by the company OpenAI.

Why executed code measures more honestly than text comparison

Many tests for AI systems compare the answer to a model solution word for word. This works poorly for program code. Two programs can look completely different and still do the same thing. Conversely, a program can closely resemble the model solution and still crash. HumanEval sidesteps this problem by simply running the code.

This made HumanEval the standard reference for years. Whenever a company introduced a new language model, the HumanEval score would almost always appear in the press release. The usual metric is called pass@1: the proportion of tasks the model solves correctly on its very first attempt. There is also pass@10, where the model is allowed to make ten suggestions and one of them has to be correct.

Progress can be clearly read from these numbers. The first Codex model from 2021 achieved around 28 percent on pass@1. Today’s top models score above 90 percent. This is precisely the problem now: a test that almost everyone passes no longer distinguishes anyone.

From function stub to test run

The procedure is the same for every task. The AI is given what is called a function stub: the name of the required function, the expected inputs, and a comment describing the task. A function is simply a named building block of a program that can be called again later. The AI fills in the missing part.

A checking program then runs the generated function with various sample values. It compares each result with what should come out. These test runs take place in an isolated environment so that faulty or dangerous code cannot harm the test machine. All tasks are written in the Python programming language.

One detail in its creation is important: the tasks were written by hand by humans, not collected from the internet. This was intentional. Had they been copied from public code repositories, they would likely already be part of the models' training material. A model could then know the solution by heart instead of working it out. Ironically, this very suspicion is now leveled at HumanEval, because the tasks have been freely available online for years.

Reading HumanEval figures in model announcements

The term is most commonly encountered in announcements about new AI models. There, bar charts appear with values like “HumanEval: 92.0%.” Such figures should be read with caution. Different providers test in different ways, for instance using multiple attempts or additional hints in the prompt, meaning the input given to the model.

A second caveat concerns the level of difficulty. HumanEval tasks are short and self-contained, usually just a few lines long. Real-world software development, by contrast, means getting to grips with a project containing thousands of files. A high HumanEval score therefore says little about whether an AI is useful in everyday professional work.

As a result, newer tests have emerged. MBPP contains similar small tasks, while SWE-bench uses real bug reports from large open-source projects. Nevertheless, HumanEval has not disappeared from the tables. It has become more of a historical benchmark by now: useful for tracking models' progress over the years, but no longer meaningful on its own.