Fill-in-the-Blank

Fill-in-the-Blank simply means “filling in gaps.” It refers to a method used to teach computer programs language. You take a perfectly ordinary text and strike out individual words. “The cat is sleeping on the sofa” becomes, for instance, “The cat is sleeping on the ___.” The program is now supposed to guess what belongs in the gap. Since the correct answer is right there in the original text, the program can correct itself – millions of times over, without any human having to step in.

Why gaps are cheaper than teachers

To teach a program something, you normally need examples that come with the correct answer attached. For recognizing photos of cats, someone would first have to label thousands of images by hand. That costs time and money. This is exactly where the trick behind fill-in-the-blank lies: the solution is already there.

Every text on the internet is automatically a practice exercise. All you need to do is cover up one word and you instantly have a question and an answer. Experts call this self-supervised learning. This made it possible, for the first time, to use gigantic amounts of text for training: entire Wikipedia dumps, book archives, news sites.

It was precisely this breakthrough that made modern language models possible. A language model is a program that understands and generates text. Without the blank-filling principle, nobody would have been able to gather enough labeled data to train such systems.

What actually happens during the guessing

The program receives the sentence with the gap and calculates a probability for a very large number of possible words. For “The cat is sleeping on the ___,” “sofa” gets a high value, “bicycle” a low one. Then a comparison is made: what actually stood in the text? If the program was off the mark, its internal settings are adjusted slightly.

These settings are called parameters. Large models have billions of them. Each individual gap shifts them only minimally. Only through billions of such corrections does something emerge that looks like language understanding. In doing so, the program doesn’t memorize grammar rules by heart. It learns which words are common in which context.

It’s important to distinguish this from a related method. In fill-in-the-blank, the model sees the text both to the left AND to the right of the gap. In next-word prediction, it only sees what comes before. The first is well suited to understanding texts, the second to writing them. ChatGPT and similar chatbots work mainly according to the second principle. The search-engine model BERT, by contrast, was trained the classic way, with gaps.

Gaps in search engines and code editors

A common misconception is that fill-in-the-blank is merely a training exercise with no practical use. In fact, it’s built into products used every day. Since 2019, Google has been deploying models trained this way to better interpret search queries. Translation services and spell checkers benefit from it as well.

The principle is especially visible in programming aids. If you’re positioned in the middle of a line in a code editor, you get suggestions for exactly that spot. The model knows the code before and after it – that is, quite literally, a gap. Providers often call this feature “Fill in the Middle.”

In technical articles and company announcements, the term usually appears in its English form, often as “Masked Language Modeling.” If you read that a model was “pretrained with masked tokens,” this is exactly what’s meant: words were covered up, and the model had to guess them. The somewhat old-fashioned term “cloze test,” by the way, comes from language teaching – you may remember fill-in-the-blank exercises from English class.