Long-Horizon Tasks

Some tasks are done with a single answer. “How many inhabitants does Hamburg have?” is such a task. Other tasks stretch over many steps that build on one another. “Plan a class trip to Rome” is one such task: checking dates, comparing prices, booking accommodation, informing parents. In AI research, such multi-step tasks are called long-horizon tasks, meaning tasks with a long horizon. The term says nothing about how hard a single step is. It only says that many steps are needed and that later steps depend on earlier ones.

Why long chains of tasks are the weak point of today’s AI

Language models are very good at individual steps. They reliably write a paragraph, translate a sentence, or fix a piece of program code. But as soon as twenty such steps are supposed to run in sequence, reliability collapses. The reason is simple probability theory: if each step succeeds 95 percent of the time, a chain of twenty steps only succeeds about 36 percent of the time. Errors don’t add up, they multiply.

On top of that, there’s a second problem. A model often doesn’t notice that it has gone astray. It consistently keeps working through a flawed plan instead of going back. Humans following a craft instruction manual usually notice after three steps that something is wrong. In such cases, an AI system sometimes builds the entire construction crooked all the way to the end.

Economically, this is decisive. An AI that writes a text on command is a tool. An AI that independently manages bookkeeping over the course of a month would be a replacement for work hours. The difference between the two is precisely the ability to handle long-horizon tasks. That’s why companies now measure how long a task may take for a model to still be able to complete it.

Planning, remembering, correcting

For a system to sustain long chains, it needs three things. First, a plan: the large task is broken down into manageable subtasks before the work begins. Second, a memory in which intermediate results are stored. Third, a control mechanism that checks whether a step has actually worked.

In practice, this is usually built as follows: the model writes down its own plan as a list and checks off items one by one. After each item, it receives feedback from the real world. For programming tasks, this is, for example, the computer’s error message. This feedback is more valuable than any self-assessment by the model, because it cannot be fooled.

A technical bottleneck is the context window, meaning the amount of text a model can keep track of at once. For very long tasks, the previous course of events eventually no longer fits. People get around this with summaries or external note files. Such abilities are increasingly trained using reinforcement learning: the system tries out long tasks and is rewarded only for the successful final result.

From agent demos to model evaluation

The term crops up everywhere AI agents are discussed. These are programs that independently operate several tools, such as a browser, a spreadsheet, and an email inbox. It is precisely there that it becomes clear whether a model merely delivers individual answers or actually completes a real task.

In news about new models, you’ll often find comparative benchmarks on this. One well-known metric is how long a task would take a human that a model can still solve in about half of the cases. This time span has been growing significantly for years, from a few minutes toward several hours. Such figures should be read with caution, because they depend heavily on the selection of test tasks.

A common mistake is to confuse long-horizon tasks with long texts. A model that summarizes a 300-page novel does process a lot of text, but in a single step. What matters is not the amount of input, but the number of dependent actions. Anyone who separates the two understands product claims from the industry considerably better.