Training Corpus

A computer program that processes language doesn’t get its abilities typed in by programmers. Instead, it reads huge amounts of material and derives patterns from it. This collected body of material is called a training corpus. The word corpus comes from linguistics, where it simply means: an organized collection of texts. In modern language programs, such a collection typically consists of web pages, digitized books, news articles, program code, and discussions from forums. You can think of the corpus as the library that a system has worked through completely, once, before it ever answers a single question.

The corpus determines what a model knows

An AI system cannot do anything that wasn’t present in some form in its data. If a corpus contains almost exclusively English texts, the result in German will be noticeably weaker. If it contains hardly any medical literature, medical answers will be superficial. The selection of data is therefore not a minor technical detail, but the single most important substantive decision of all.

Whatever goes wrong is likewise passed on. If the corpus predominantly contains texts in which certain professions are attributed only to men, the model adopts this skew. Experts then speak of bias, meaning a systematic distortion. Plainly false claims from the internet also end up in the corpus and can later resurface as an answer.

On top of this comes a legal point of contention. Many texts and images online are protected by copyright. Whether they may be included in a corpus without permission is, in several countries, the subject of court proceedings. Publishers, music labels, and photo agencies have therefore filed lawsuits against AI companies.

From raw data mountain to usable collection

At the start there is usually an automated scan of the internet. Programs called crawlers follow links and save page contents. The result is an enormous, very messy pile of data. A large part of it is unusable: ad blocks, navigation bars, spam, automatically generated nonsense.

That’s why filtering follows in several stages. Duplicate texts are removed, because learning identical passages thousands of times distorts the result. Pages of very low quality are thrown out, often with the help of a small auxiliary program that rates text. Personal data such as addresses or phone numbers are searched for and deleted. In the end, often only a small fraction of the original data volume remains.

The size of a corpus is usually measured in tokens. A token is a word fragment, roughly three-quarters of an average word. Large language models today are trained with several trillion tokens. For comparison: the entire German-language Wikipedia is in the range of a few billion tokens, so it’s only a tiny fraction of that. A common misconception is that more data is always better. Past a certain point, clean, diverse material yields considerably more than sheer volume.

When the corpus makes the news

Publicly known collections carry names like Common Crawl, a freely available archive of billions of web pages, or The Pile, a curated text collection for research purposes. When reports state that a model was trained on fifteen trillion tokens, this refers to the size of its corpus. For commercial systems, however, the exact composition often remains a trade secret.

You also encounter this topic in everyday life. If a chatbot knows nothing about events from the last few months, that’s due to a cutoff date in the corpus. After that date, no more data was collected. And when platforms like Reddit or newspaper publishers sign agreements with AI companies, this is exactly what it’s about: paid access to material for future training collections.