Self-Supervised Learning

For a computer program to learn something, it needs practice tasks with known solutions. In the past, humans entered these solutions by hand: they marked on thousands of photos where a cat could be seen. That is expensive and slow. In Self-Supervised Learning, the program instead generates its own practice tasks from the raw data. A typical trick: you hide a word in a sentence, and the program has to guess which one was missing. The correct answer is already right there in the text — nobody has to write it down separately.

Why masses of text from the internet suddenly became usable

Labeled data is scarce, unlabeled data exists in almost unlimited amounts. The internet contains billions of pages of text, plus images, videos, and audio recordings. As long as a method required human labeling, this mountain of data could barely be exploited. Self-Supervised Learning removed exactly this bottleneck.

This is the main reason why such large language models have existed since around 2018. Systems like ChatGPT were trained on amounts of text a human could not read in a thousand lifetimes. No company would ever have had the money to annotate these texts by hand. The detour via self-generated tasks was the precondition for this leap.

Even more important is a side effect: anyone who learns to predict the next word must, in the process, know a great deal about the world. To complete the sentence “The capital of Japan is ...” requires factual knowledge. To complete “He stumbled and fell to ...” requires everyday logic. Out of what seems like dull guesswork, a broad model of the world emerges as a byproduct.

Fill-in-the-blank texts as a training task

The basic principle is always the same: part of the data is hidden, the rest remains visible. The model is supposed to reconstruct the hidden part. Afterward, the prediction is compared with the actual value and the model’s internal settings are adjusted slightly. These settings are called parameters; large models have billions of them. The process repeats itself billions of times.

With language models, there are two common variants. One hides individual words in the middle of a sentence, which is how the BERT model worked, for instance. The other shows only the text up to a certain point and has the continuation predicted; this is what the GPT models are based on. With images, parts are cut out, sections are rotated, or the model is made to recognize whether two sections come from the same photo.

A common misconception is that Self-Supervised Learning is the same as unsupervised learning. In unsupervised learning there is no target answer at all; the program merely searches for patterns and groups in the data. In Self-Supervised Learning, there is indeed a clearly correct answer — it just comes from the data itself instead of from a human. It can be described as supervised learning with an automatically generated answer key.

From base model to finished product

In news coverage about AI, this method usually appears under the term pretraining. This refers to the first, expensive phase in which a model learns in a self-supervised manner on huge amounts of text. The result is called a base model. It can do a lot, but it is not yet a helpful conversation partner. Only afterward comes a smaller phase with human-created examples and ratings that makes the model polite and useful.

Even outside of chatbots, the principle is in use everywhere. Speech recognition on phones, automatic translation, medical image analysis, and the recommendations in streaming services all use self-supervised pretrained models. The advantage is always the same: in the end, only a few labeled examples are needed for the specific task.

For the debate about copyright, this point is central. Because the method can exploit arbitrary raw data, books, newspaper articles, and photos were fed into training without consent. This is precisely what several major lawsuits against AI companies currently revolve around.