Long-Horizon Coherence

Programs like ChatGPT generate text by repeatedly predicting the next word. Over a few sentences, this works astonishingly well. Over hundreds of pages, or over a conversation that stretches out across weeks, it becomes difficult. The program then contradicts itself: a character is suddenly given a different name, a number mentioned earlier no longer matches, a decision that was made is silently overturned. Long-horizon coherence is the technical term for the ability to avoid exactly that. What is meant, then, is the question of whether a system sticks to one line over long stretches, rather than starting fresh in every section.

The difference between a good page and a good book

Today’s AI can often write short texts at a very high level. An email, a summary, a piece of program code: this regularly succeeds convincingly. But as soon as the task grows larger, quality collapses. A novel needs a plot that still holds up after 300 pages. Software needs building blocks that still fit together even at the twentieth step of work.

Economically, this point is decisive. Many valuable tasks are, after all, long tasks. A tax audit, a research project, looking after a customer over months: all of this consists of many steps that build on one another. A system that only masters individual steps constantly needs a human as a supervisor. Only with reliable long-horizon coherence could it take over entire processes.

That is why experts nowadays measure pure answer quality less. Instead, they ask: how long can a system work independently before it goes off the rails? This span of time has noticeably lengthened in recent years, moving from minutes toward several hours. But it is still the bottleneck.

Why models lose the thread

A language model has no memory in the human sense. With each answer, it only sees what is currently in its working memory. This area is called the context window and holds a limited amount of text. Once it is full, older material drops out or is heavily shortened. Whatever drops out simply no longer exists for the model.

Larger context windows only partially solve the problem. Some models today process a million words at once. Nevertheless, the hit rate declines the further apart pieces of information lie. Experts speak of the loss in the middle: at the beginning and end of the text, the model reliably finds details, but in between it overlooks them more often.

On top of that comes a second effect. Every small error remains in the text and serves as the basis for the following sentences. Errors thus accumulate rather than cancel each other out. A common misconception is therefore to regard long-horizon coherence as a purely memory-related problem. It is also a planning problem: the model would need to recognize that it is currently deviating from its own plan.

Countermeasures: notes, summaries, interim checks

Developers therefore give systems an external memory. Important facts are stored in a database and retrieved back into working memory as needed. Other approaches regularly summarize long conversations and carry forward only the summary. In programming tools, this can be seen in files where the system notes down its own rules and interim states.

In everyday life, one usually encounters the term indirectly. If a chatbot forgets your name after twenty minutes, that is a coherence problem. If an AI video changes a person’s facial features from second to second, that is one too. In product announcements, buzzwords like Memory, Projects, or Agent crop up for this.

In the news, it is therefore worth asking a skeptical question. When a company talks about AI employees, one should want to know how long these hold up without correction. This is precisely what determines whether a tool merely delivers impressive individual answers or truly handles entire tasks.