Long-Context Inference

An AI language model doesn’t read text like a human does, page by page. It processes everything at once — within a fixed frame known as the context window. This window determines how much text the model can hold “in view” simultaneously. Long-context inference is the process in which this window is very large: instead of a few paragraphs, the model can process entire books, lengthy contract texts, or hours-long conversation transcripts all at once. Older models managed a few thousand words; current systems process millions of them. This may sound like simply more of the same, but technically it’s a fundamentally different problem.

Why the length of the context window matters so much

Most practical tasks fail not due to a lack of knowledge, but due to too little space. Anyone asking a model to summarize a hundred-page contract needs to be able to input the entire contract at once. With a small context window, only a workaround remains: cutting the text into pieces, processing each part separately, and manually assembling the results. In doing so, connections spanning the entire document get lost.

A larger context window fundamentally solves this problem. A model that sees the entire contract can link a clause on page 3 to an exception on page 87. This is a qualitative difference, not a gradual one. That’s why context length, alongside sheer model size, is considered one of the most important factors when comparing modern language models.

The scaling problem behind long contexts

The core technical problem is the so-called attention mechanism — the procedure by which a language model decides which parts of the text are relevant to one another. This procedure costs computing time that grows quadratically with text length. Doubling the input length roughly quadruples the computational effort. At a million words, this simply becomes unaffordable unless it is mitigated.

Researchers have developed several ways to slow this growth. One method has the model focus on only a sliding portion of the text at a time, rather than everything. Another compresses older parts of the conversation into a kind of short note. Still others replace the classic attention mechanism with methods that scale more favorably mathematically. None of these approaches comes for free: every simplification risks the model overlooking details located far back in the text.

A well-known test problem from research is called “Needle in a Haystack.” In it, a single piece of information is hidden somewhere in the middle of a very long document, and the model is asked about it. It turns out that many models get worse as context grows, especially when the answer is hidden in the middle of the text rather than at the beginning or end. This phenomenon is called “lost in the middle” and remains unsolved to this day.

Long-context inference in products and headlines

Google's Gemini models have turned the context window into a publicly debated competitive topic. Gemini 1.5 Pro, at its unveiling in 2024, supported up to one million tokens — a unit roughly equivalent to half to three-quarters of a word. That corresponds to about 700,000 words, or the content of several thick novels. Anthropic’s Claude and OpenAI’s GPT-4 models followed suit with their own long context windows.

In practice, long-context inference shows up anywhere large amounts of text need to be automatically evaluated. Lawyers upload entire sets of contracts, developers have their whole code repository analyzed, journalists query a model about a large document collection. AI assistants meant to remember long conversations also depend on this capability.

An important point here is a common confusion: a large context window is not the same as a good memory. The model doesn’t store anything permanently — once the conversation ends, everything is gone. What long-context inference provides is attention span, not long-term memory. True recall across many conversations requires other methods, such as external databases in which relevant information is stored and retrieved as needed.