Assistance Games

An assistance game is a formal model — a kind of mathematical rulebook — that describes the relationship between an AI and a human. The AI has a task: it is supposed to help the human achieve their goals. The problem is that the AI doesn’t know these goals. It has to infer them from the human’s behavior. This sounds obvious, but it is a crucial difference from classical AI systems, which simply optimize a fixed task — regardless of what the human actually wants.

Why classical goal optimization is dangerous

If you give an AI a fixed goal, it will optimize that goal as relentlessly as possible. This sounds good, but it leads to a well-known problem: the specified goal almost never matches perfectly what the human actually means. An example: you program an AI to maximize points in a game. It might find a way to exploit the game instead of playing it — technically correct, but completely wrong.

This mismatch between the stated goal and the actual wish is called the alignment problem in research. Assistance games are an attempt to solve it at a fundamental level. Instead of programming a goal into the AI, you give it a task: figure out what the human wants, and help them with it.

The principle: uncertainty as a safety feature

In the model of an assistance game, both sides — human and AI — know that the AI does not know the human’s goals. The AI does not treat this ignorance as a flaw, but as a starting point. It observes what the human does and infers from this what matters to them. At the same time, the AI acts in a way that minimizes the harm it could cause before it knows for certain what the human wants.

This leads to something surprising: an AI operating according to the assistance game model is inherently cautious. It asks before doing something irreversible. It allows itself to be switched off, because it understands that being switched off might matter to the human. This property — allowing oneself to be corrected voluntarily — is considered a central requirement for safe AI systems.

The model was developed primarily by computer scientist Stuart Russell and introduced to a broad audience in his book “Human Compatible” (2019). Mathematically, it builds on so-called inverse reinforcement learning — a method in which an AI infers goals from observed behavior, rather than having goals directly programmed in.

Assistance games in research and product development

So far, assistance games are primarily a theoretical tool. Research groups at universities such as Berkeley and Oxford use the model to formulate AI safety questions precisely. Anyone wanting to show why an AI should keep itself shutdownable can prove this mathematically using assistance games — rather than merely demanding it intuitively.

In products, the concept so far appears only indirectly. Modern language models such as ChatGPT or Claude are trained using methods influenced by similar ideas: they are supposed to listen to human feedback and adjust their behavior instead of stubbornly pursuing a fixed goal. This is not a complete assistance game, but it is the same underlying idea.

In news articles, the term often appears in the context of AI safety debates. When researchers warn that current AI systems are poor at understanding human values, they often mean exactly the problem that assistance games are meant to solve. The term is a sign that the discussion on this topic is becoming mathematically concrete — rather than remaining merely philosophical.