ARC-AGI-3

ARC-AGI-3 is a test for computer programs that are supposed to solve problems on their own. Such tests are called benchmarks: standardized tasks that all participants receive so that their results can be compared. In ARC-AGI-3, the task consists of playing small games on a colored grid. The rules of these games are explained nowhere. The program has to figure out through trial and error what its moves do and how to win. The test is being developed by the ARC Prize Foundation, a nonprofit organization built around researcher François Chollet.

The name stands for “Abstraction and Reasoning Corpus.” Translated, that means roughly: a collection of tasks for abstraction and reasoning. The number 3 marks the third generation. The first two versions showed only silent picture puzzles. Version 3 is the first in which the program has to act and observe the consequences of its actions.

The dispute over what machines can really do

Most well-known AI tests consist of exam questions. Math problems, medical exams, coding tasks. Such questions appear millions of times over on the internet. A program trained on huge amounts of text has often already seen very similar tasks. A good result then proves little about genuine understanding.

ARC-AGI-3 is deliberately built so that memorization does not help. Every game is newly invented and appears in no training dataset. Anyone who wants to score here must derive a rule from just a few observations. Many researchers consider exactly this ability to be the core of intelligence.

Also important is the point of comparison: humans. The test tasks are played beforehand by test subjects, and only what practically every human can manage is recorded. This makes the gap between human and machine directly visible. In the predecessor version, humans were far ahead for a long time, while good programs solved only a small fraction. Such figures serve as a counterweight to providers' marketing promises.

Playing instead of guessing: how the tasks are structured

A task in ARC-AGI-3 looks like a very simple video game. Colored tiles sit on a grid. The program can make moves, such as moving a figure or clicking a tile. After each move, it gets the new image back. There is no more information than that: no instructions, no text, no explanation of scoring.

The process is thus a loop of observing, acting, and adjusting. The program forms a hypothesis about the rules. Then it tests this hypothesis with a move. If the result doesn’t fit, the hypothesis has to be corrected. You can picture it like a board game without a rulebook: you play a card, see the other players' reaction, and infer the rule from that.

What is measured is not just whether a game is solved. How many attempts are needed for that is also important. This measure is called efficiency. A system that blindly tries a million moves is not considered intelligent. In addition, the organizers cap the computing cost per task. This is meant to prevent sheer computing power from taking the place of thinking.

ARC-AGI-3 in headlines and company presentations

The test mainly turns up in the news when a company unveils a new model. Models are the trained programs behind services like ChatGPT or Gemini. Their results on ARC-AGI are given as percentages and are readily cited. Headlines like “Model X reaches Y percent for the first time” often refer to this benchmark.

Around it there is an open competition, the ARC Prize. Research teams and individuals submit solutions there, with prize money on offer. The results are publicly listed in a leaderboard. This also makes the test interesting to investors who want to know whether progress is real or merely claimed.

A common misconception: a top score on ARC-AGI-3 does not mean that a machine thinks like a human. The test measures a narrowly defined slice, namely the rapid inference of new rules. It does not test language sense, factual knowledge, or everyday reliability. It should therefore be read as one of several thermometers, not as an overall grade.