Memory Decay

Computer programs that understand and write text do not remember everything equally well. What was said at the beginning of a long conversation is often reproduced less accurately than what was said most recently. This gradual loss is exactly what is called memory decay. The term is used in two senses. One concerns short-term retention within an ongoing conversation. The other concerns long-term learned knowledge that gets overwritten during retraining.

Why chatbots lose the thread mid-conversation

For users, memory decay is the cause of a very typical frustration. At the start, you explain to an assistant program exactly how the answer should look. Twenty messages later, it no longer sticks to that. The program is not being inattentive—the earlier information has simply lost weight.

Economically, this is a serious problem. Companies want to use AI systems for long tasks, such as handling a customer case over several weeks. If the system keeps forgetting earlier agreements along the way, it becomes unusable. That is why providers invest a lot of money in longer and more stable remembering.

A second point concerns the developers themselves. Anyone who trains a finished model on a new task risks it losing old abilities. A language model that is intensively retrained on programming code may afterward write poetry worse. Experts refer to this as catastrophic forgetting.

Context windows, weighting, and overwritten numbers

The so-called context window is responsible for short-term memory. This is the amount of text a model can take into account at once when generating an answer. It is measured in tokens, i.e., word fragments. Modern models can handle hundreds of thousands of them. Once the window is full, the oldest text falls out and is gone for good.

But decay already sets in before this hard cutoff. With every answer, the model evaluates which passages of text are important. Information that is nearby and repeated multiple times is given more weight. Details that are far away are weighted more weakly and barely have any effect anymore. You can picture it like a chalkboard that is constantly being written on anew, while older writing gets smudged away.

Long-term memory works differently. A model’s knowledge is stored in billions of numbers, the parameters. New training changes precisely these numbers. If a model is trained heavily on a single domain, values that previously carried other knowledge get shifted. Countermeasures include smaller learning steps or training methods that touch only a small portion of the parameters.

From the assistant’s sticky note to contract review

In everyday life, memory decay is encountered by anyone who has longer chats with an AI assistant. A practical way to deal with it is to repeat important instructions regularly. Summarizing the conversation so far also helps, because it brings the key points back fresh into the window.

The providers themselves work with additional systems. Many assistants store selected facts about the user in a separate file, often simply called memory. Before each answer, matching entries from it are inserted into the text. A related method searches for information in documents and supplies only the relevant passages. Neither replaces genuine memory, but both compensate for its weakness.

In tech news, the topic usually comes up on two occasions. First, when manufacturers advertise ever-larger context windows. Second, when tests show that a model, despite having a huge window, overlooks information in the middle of a long document. A common misconception is therefore the assumption that a large window automatically means reliable remembering. Capacity only tells you how much fits in, not how well it is used.