LiveCodeBench

LiveCodeBench is a collection of programming tasks used to test computer programs that write program code themselves. Such programs learn from huge amounts of text and code from the internet and can then formulate a solution on command. To check how good they really are at this, they are given tasks and the proposed solution is then run by a computer. If the program runs through and delivers the correct result for all test cases, the task counts as solved. The proportion of solved tasks is the score. The word “live” in the name means that fresh tasks are constantly added as soon as they are published on the web.

Why fresh tasks make the difference

Older programming tests like HumanEval consist of a fixed list of tasks. These lists have been publicly available on the internet for years. When a model is trained with data from the web, the tasks along with sample solutions often end up directly in the training material. The model has then memorized the answer instead of working it out. Experts call this problem contamination, meaning pollution of the test data.

LiveCodeBench gets around this with a simple trick. The publication date is stored for each task. This means the test can be restricted to tasks that only appeared after a model’s training cutoff. What the model cannot have seen, it must actually solve. If the score drops significantly on these new tasks, the previously high value was probably partly memorized.

This is relevant for companies and investors because benchmark results play a big role in the marketing of new models. A number from a contaminated test says little about real-world practice. LiveCodeBench is therefore considered one of the more meaningful benchmarks for programming ability.

Where the tasks come from and what is measured

The tasks come from competitive programming platforms, mainly LeetCode, AtCoder, and Codeforces. New puzzles are regularly published there, for example: Find in a list of numbers the longest stretch whose sum does not exceed a certain limit. Each task comes with hidden test cases, i.e. example inputs with expected output. LiveCodeBench’s collection system automatically pulls in new tasks and tags them with their publication date.

The testing procedure is always the same. The model receives the task description as text, writes a program, and this program is executed in a sandboxed environment. There is no partial credit: either all test cases pass or the task counts as unsolved. Usually the model gets several attempts, and it is noted whether at least one of them was correct.

In addition to pure code writing, LiveCodeBench also tests three further abilities. The model should repair its own code when a test case fails. It should predict what output a given program produces. And, conversely, it should find an input that leads to a specific output. These additional parts show whether a model truly understands code or is merely assembling matching text fragments.

LiveCodeBench in model announcements

Anyone reading the presentation of a new AI model will usually find a table with benchmark values. LiveCodeBench appears there almost always alongside tests for mathematics and general knowledge. A percentage value is given, often together with a time period such as “tasks from August 2024 to January 2025”. This time period matters: two numbers are only comparable if they refer to the same set of tasks.

A common mistake is to conclude directly from a good score that a model is a good programming assistant. Competitive tasks are cleanly described, short, and have a clear solution. In everyday professional work, by contrast, it’s about grown codebases with thousands of files, unclear requirements, and old bugs. Other tests exist for that, such as SWE-bench, which uses real bug reports from open-source projects.

In practice, you mainly encounter LiveCodeBench in news and product announcements. The value is sensibly read as an indication of how well a model cracks well-defined programming problems. A jump from 40 to 70 percent within a year is a strong signal. A lead of two percentage points between two models, on the other hand, says hardly anything.