Pass@1

Pass@1 is a metric from tests for AI systems designed to solve tasks. The program is given a task and made to generate exactly one answer. That answer is then checked: correct or incorrect. Repeating this across many tasks yields a percentage. A Pass@1 of 70 percent means: for 70 out of 100 tasks, the first attempt was correct. The name comes from English and translates loosely to “passed on the first try.”

Why the first attempt matters

In practice, one often only gets a single attempt. Anyone asking an AI for a formula or a piece of software code wants a usable answer, not a lottery. Pass@1 measures exactly this everyday scenario. That’s why this number appears in almost every press release about a new model.

There are also variants such as Pass@10 or Pass@100. There, the program is allowed to propose ten or a hundred solutions, and it is enough if one of them is correct. These values are always higher than Pass@1, sometimes dramatically so. A model might reach 40 percent on Pass@1 and 90 percent on Pass@100. This means: the necessary knowledge is present in the model, but it unreliably picks the right choice.

For companies, this difference is worth real money. A hundred answers cost a hundred times the computing time. A high Pass@1 value therefore directly lowers the operating costs of a service.

From test case to percentage

Pass@1 is most commonly measured on programming tasks. A well-known task set is called HumanEval and contains just under 165 small programming problems. Each task comes with ready-made check routines, so-called tests. They call the written code with example values and compare the result to the expected output. If everything passes, the task counts as solved.

Importantly, it is not evaluated whether the code looks elegant or resembles the model solution. Only behavior decides. This makes the measurement objective, but also coarse. A program can pass all tests and still fail on other inputs.

One technical detail often causes confusion. Language models do not answer the same way every time; they contain a random component. That’s why researchers usually generate several answers and calculate from them the expected value for a single attempt. Pass@1 is therefore normally a statistical average, not the result of a single run.

Reading Pass@1 in model announcements

When a company introduces a new model, Pass@1 can be found in the tables in the appendix. There you’ll find names like HumanEval, MBPP, or SWE-bench, each paired with a percentage value. These numbers are the reason trade media write about a model being “better at coding.” Online rankings, so-called leaderboards, also sort models by such values.

These numbers should be read with caution. Task sets like HumanEval are public and available online. If parts of them end up in a model’s training material, it already knows the solutions. Experts call this contamination. The measured value is then too high and says little about genuine capabilities.

Furthermore, Pass@1 only applies to tasks with a clearly verifiable result. This works well for mathematics or code. For an essay, a translation, or advice, there is no automatic right or wrong. There, one relies on human evaluations or comparisons between two answers. Pass@1 is thus a precise metric for a fairly narrow, but economically very important, domain.