Active Memory

When an AI model works on a long task – for example summarizing a multi-page text or carrying on a conversation across many messages – it needs to remember things along the way. Active Memory is the area where exactly this happens: it holds the information the model needs at that moment. You can picture it like a notepad you write on during an exam and then throw away afterward. Once the task is done, the content is gone – it is not stored permanently. What matters is how much fits on this notepad and how quickly the model can access it.

Why the size of Active Memory determines capability and limitations

A model with a small Active Memory loses track during longer tasks. For example, it may no longer be able to take the beginning of a long document into account once it has reached the end. This is not a flaw in training but a technical limit: the memory is simply full.

In practice, this memory is often constrained by what is called the context length – that is, the maximum amount of text a model can process at once. Early language models could handle only a few thousand words. Newer models now manage hundreds of thousands of words, roughly the length of several novels. The larger the Active Memory, the more complex the tasks a model can solve in a single pass.

At the same time, a larger memory costs more computing power. Every additional word in the Active Memory must be related by the model to every other word. As a result, the computational effort grows not linearly but disproportionately with the length of the content.

How Active Memory is implemented in modern AI systems

In large language models – that is, AI systems that process and generate text – the Active Memory is closely tied to what is called the context window. Everything within this window is “visible” to the model and can influence the response. Whatever lies outside of it does not exist for the model at that moment.

Some systems supplement this short-term memory with external tools: the model writes important intermediate results to a database and retrieves them again when needed. This allows it to also work with information that doesn’t fit into the context window. This approach is often called Retrieval-Augmented Generation, or RAG for short. The Active Memory nevertheless remains the central workspace – the database is merely a supporting tool around it.

There are also research approaches in which the model itself decides which information to keep in Active Memory and which to discard – similar to a person deciding during a long phone call what they really need to remember. These methods are not yet standard, but are being studied intensively.

Active Memory in products and headlines

Anyone who uses ChatGPT, Gemini, or similar chatbots encounters Active Memory in every conversation. When a chatbot suddenly forgets what was said at the beginning of a conversation after many messages, that’s not a coincidence – the context window was full, and older messages were pushed out. Some services even explicitly display how much of the available memory is already in use.

In tech news, the term frequently comes up when new models with record-breaking context lengths are introduced. Google's Gemini 1.5 Pro, for instance, was promoted in 2024 as being able to process up to one million so-called tokens – roughly speaking, word fragments – at once. That corresponds to about 700,000 words. Such figures are direct statements about the size of the Active Memory.

The topic is also relevant in enterprise applications. Companies that use AI to analyze long contracts or evaluate meeting minutes need to know how much text a model can take in at once. If a document is longer than the Active Memory, it has to be split up – which can affect the quality of the analysis.