Training corpus

A computer program meant to process language or images doesn’t get its abilities typed in by programmers. Instead, it is presented with a very large number of examples and searches for patterns in them on its own. The entire collection of these examples is called a training corpus. For systems that write text, it consists of web pages, books, newspaper articles, forum posts, and program code. The scale is hard to imagine: modern collections contain several trillion words, more than a human could read in a thousand lifetimes. The word corpus comes from linguistics, where it simply means an organized collection of texts.

The corpus determines what the model can do

A system learns exclusively from what it has been shown. If a language is almost entirely missing from the corpus, the finished model will speak it poorly. This is precisely why many AI systems appear noticeably more fluent in English than in German or Finnish. The English share is by far the largest in almost all major collections.

The same applies to biases in content. If the texts predominantly show men as engineers, the model adopts this association as a pattern. Experts then speak of bias, meaning a systematic skew. This skew is not a programming error but a faithful reflection of the underlying data.

This is also why the corpus is legally sensitive. Newspaper publishers, book authors, and photo agencies have sued companies like OpenAI because their works ended up in training collections without permission. These proceedings are among the most important legal disputes in the industry. This matters to investors because a ruling could make entire business models more expensive.

From raw download to filtered collection

At the start there is usually a scrape of large portions of the open internet. Such raw data, however, is unusable if applied directly. It contains ad banners, navigation bars, spam, and machine-generated nonsense. A large part of the work therefore consists of filtering and sorting out unwanted material.

An important step is called deduplication: identical or nearly identical passages of text are removed. Otherwise the model would see the same paragraph a thousand times and memorize it instead of generalizing. In addition, companies filter out depictions of violence, personal data, and legally problematic content. What remains is often only a small fraction of the original material.

A common comparison is with cooking: the corpus is the list of ingredients, the training is the recipe. Bad ingredients cannot be saved by any recipe. This is precisely why the focus of research has shifted. Instead of collecting ever more data, labs are increasingly paying attention to data quality. Smaller, carefully curated collections sometimes deliver better results than huge unfiltered ones.

Why newspapers and Reddit are suddenly signing contracts

In business news, the topic usually appears in the form of a licensing deal. Reddit, the news agency AP, and several publishers have signed contracts with AI companies. What is being sold is access to their content as training material. Such deals have by now reached a scale of several hundred million dollars.

Politics is also getting involved. The European Union’s AI Act requires providers of large models to publish a summary of the training data used. So far, most companies keep their exact compositions secret because they see them as a competitive advantage. A common misconception, by the way, is that the model stores its corpus like a database. In fact, only the learned patterns remain; the original texts are not stored in the finished model.

As a user, you notice the corpus wherever a system hits its limits. If you ask about an event from yesterday, it often won’t know the answer. The reason is the so-called knowledge cutoff: no data was collected after a certain date. The model only receives current information if it is additionally allowed to look things up online.