Parametric Knowledge

A language model like ChatGPT is fed vast amounts of text. In doing so, it internally adjusts millions to trillions of numbers so that it can continue texts well. These numbers are called parameters. Everything the model can answer after training, without looking anything up anywhere, resides in exactly these numbers. This stock is called parametric knowledge. You can think of it like knowledge a person carries in their head: available without needing to look it up, but also no longer correctable once it has been learned incorrectly.

Why the model still needs books

Parametric knowledge has a fixed cutoff date. It ends where the training data ends. A model whose data was cut off in the summer of 2024 doesn’t know any election results from the fall of 2024. But it often doesn’t know that it doesn’t know them. This is exactly where many fabricated answers come from, which the industry calls hallucinations.

A second problem is accuracy. The model doesn’t store texts verbatim, only statistical patterns. Frequent facts stick firmly, rare ones blur. The birthplace of a famous person will usually be correct, but the revenue of a mid-sized supplier in 2019 probably won’t be. The model sounds equally confident in both cases.

That’s why companies supplement their models with external sources. The best-known method is called RAG, short for retrieval-augmented generation: before answering, the system searches for matching documents and presents them to the model as text. Experts then speak of non-parametric knowledge. It is current, verifiable, and can be swapped out without retraining the model.

How facts migrate into the weights

During training, the model receives texts in which individual words are missing or need to be continued. If it guesses wrong, the parameters are shifted a tiny bit. This process repeats trillions of times. Facts that appear in similar form across many texts become especially deeply ingrained.

Importantly, there is no storage cell for a single fact. The knowledge is distributed across many parameters, and one parameter contributes to many different facts. Research shows that certain layers of the model contain particularly large amounts of factual knowledge. But a fact cannot simply be deleted like a row in a spreadsheet.

This is exactly what makes corrections difficult. There are methods known as model editing that rewrite individual statements in the model in a targeted way. They work, but often bring side effects elsewhere. The usual approach instead is retraining with new data or presenting correct documents at runtime.

The term in products and headlines

In everyday use, you notice the difference by the source citation. If a chatbot answers without links, the answer usually comes from parametric knowledge. If it shows footnotes or web pages, it has additionally searched. Perplexity and Google's AI Overviews rely heavily on search, while classic chat models more often answer from memory.

In companies, the term comes up in data protection issues. If personal data ended up in training, it is embedded in the parametric knowledge and is hard to remove. The same applies to copyrighted texts, which are currently the subject of several lawsuits. That’s why companies prefer to put corporate knowledge into a document collection rather than into a newly trained model.

A common misconception, by the way, is that the context window is part of this knowledge. The context window is just the text you just entered, and it’s gone again after the conversation. Parametric knowledge, by contrast, remains unchanged in the model, no matter how often you talk to it.