LoCoMo

LoCoMo stands for “Long-Context Conversations with Memory” and is a benchmark – a standardized testing course for AI models. It checks whether a language model keeps track of the thread in very long conversations. This doesn’t mean short chats, but conversations spanning dozens of sessions and simulating months of time. What’s special: the conversations in the dataset come from real recordings between humans and were not artificially simplified. A model that performs well on LoCoMo can therefore still recall details from the first conversation after 50 simulated days.

Why a long conversation history overwhelms models

Language models work with a so-called context window – a limited memory area in which they retain the conversation history so far. The longer a conversation becomes, the more older content falls off the end or is simply ignored by the model. This is not a software bug, but a fundamental technical limitation.

LoCoMo makes this problem visible and measurable. Before the benchmark, it was hard to compare which model truly performed better in long conversations – because there was no common yardstick. With LoCoMo, researchers can specifically ask: Does the model remember a detail from week three when the conversation is now in week twelve? Can it be confused by contradictory later statements? These questions are especially important for applications where reliability over time matters.

Structure of the test and what it assesses

The dataset contains long conversation histories – sometimes with over 300 conversation turns – compiled from real human conversations. Concrete questions were derived from these that the model must answer. These questions fall into various categories: direct recall of individual facts, temporal ordering of events, recognizing contradictions, and drawing conclusions from multiple statements at once.

The difficulty here is that the decisive information is often located far from the end of the conversation. A model that only processes the “fresh” part of the conversation automatically fails. LoCoMo thus does not measure general language competence, but specifically memory over long periods of time.

The evaluation uses automatic metrics – that is, formulas that compare the model’s output with the correct answer – but also assessments by other AI systems in order to fairly judge open-ended answers. Both together give a more realistic picture than a pure yes-no test.

LoCoMo in research and product development

LoCoMo was published by researchers at Meta AI. Since then, it has appeared in research whenever teams want to test new methods for improving models' long-term memory. Typical approaches compared against LoCoMo include external memory – essentially a kind of notebook that the model fills and queries in parallel – or compressed summaries of older conversation segments.

In product development, LoCoMo is relevant for all applications based on long user profiles: personal assistants that are supposed to remember preferences, therapy apps with conversation history, or customer service bots that need to know a user’s history. Anyone building such a product wants to know whether their model still knows the first conversation after a hundred conversations.

LoCoMo is thus not an end product, but a tool. It forces AI research to take seriously a problem that remains invisible in short demos: that a model may answer impressively, yet in the very next conversation has already forgotten who you are.