Model Context
The model context is the amount of text an AI language program has in view at once when handling a request. Anything that doesn't fit cannot be taken into account by the program when generating its response.
Programs like ChatGPT generate text by predicting, word by word, what fits next. To make this prediction, they look at everything currently in front of them: your question, uploaded documents, the conversation so far. This entire body of text is called the model context. It is limited, specifically to a fixed number of text building blocks. Whatever doesn’t fit simply doesn’t exist for the program at that moment. You can picture the model context as a desk: only what’s lying on it can be used while writing.
Why the desk eventually fills up
The size of the context is measured in tokens. A token is a piece of text, usually a short word or a syllable. As a rough rule of thumb: 1,000 tokens correspond to about 750 English words. Early chatbots got by with around 4,000 tokens, roughly ten pages of a book. Today’s top models handle several hundred thousand tokens—entire books.
This limit determines which tasks are even possible in the first place. A model with a small context can’t review a 200-page contract in one go. Nor can it carry on a long conversation without forgetting the beginning. This is exactly why providers advertise large context windows so prominently.
A common misconception: context is not memory. It’s more like a short-term buffer that gets refilled anew with every request. When a chatbot seems to remember an earlier conversation, this is usually because the software automatically copies old notes back into the context. The model itself learns nothing new from your messages.
What happens when it overflows
Once the limit is reached, something has to give. Many chat interfaces then silently delete the oldest messages. Users often don’t notice, but they do wonder why the program suddenly mixes up details from the start of the conversation. Other systems summarize the older history beforehand to save space.
Large contexts are also expensive. When computing, the model in principle compares every piece of text with every other piece. Doubling the amount of text roughly quadruples this effort. That’s why long requests cost noticeably more and take longer with commercial providers. Newer methods ease this problem, but they don’t eliminate it.
A large context also doesn’t guarantee good use of it. Studies show an effect known as “Lost in the Middle”: information at the beginning and end of a long text is found more reliably than information in the middle. So a model with a context of a million tokens doesn’t automatically read everything with equal attention.
Context in chatbots, documents, and coding tools
In everyday use, you run into this limit fastest when uploading files. If a chatbot rejects a PDF or only partially processes it, the context is usually too small. This also matters in programming: assistants like GitHub Copilot have to decide which files of a project to include at all.
In product announcements, context sizes are a key selling point. When an announcement talks about a model “with a context of a million tokens,” this upper limit is exactly what’s meant. Such numbers allow rough comparisons between providers, but they say nothing about the quality of the responses.
Because context is scarce and expensive, a whole technique has grown up around it. With RAG, short for Retrieval-Augmented Generation, an additional system first searches a database and places only the relevant text passages into the context. This way, a company can make thousands of documents usable without feeding them all in at once. This selection work is today often more important than the raw size of the context window.