Retrieval

Retrieval is the process of searching out matching content from a large collection of documents. Someone asks a question, and a search system fetches the few text passages that presumably contain the answer. Everyone knows this principle from a search engine: you type something in and get back a list of results. What’s new is that today this list of results is often no longer intended for humans at all. Instead, the found text passages travel directly to a program that formulates an answer from them in complete sentences. Retrieval, then, is the search step that precedes a machine-generated answer.

Why language models guess without retrieval

A language model is a program that has learned to write meaningful text. Its knowledge comes from the texts it was trained on. However, this knowledge is not stored in the model as a list of facts, but only as a rough pattern. When a detailed question comes up, the model, in case of doubt, invents a plausible-sounding answer. Experts call this hallucination.

There is a second problem on top of this: training ends at a certain point in time. Anything that happened after that is unknown to the model. A model from 2024 knows nothing about last week’s quarterly report. Retraining would be the most expensive conceivable way to supply such information afterward.

Retrieval solves both problems at once. The document collection can be updated at any time without touching the model. And because the answer is based on concretely found text passages, the system can cite its source. That’s exactly why retrieval is so appealing to companies: in many fields, an answer without a verifiable source is worthless.

From question to results list

First, the document collection is broken down into small pieces, usually a few paragraphs long. Each piece is translated into a long sequence of numbers, a so-called embedding. This sequence of numbers describes the meaning of the text. Texts with similar meaning receive similar sequences of numbers. All these sequences of numbers end up in a special database.

When a question comes in, it too is translated into such a sequence of numbers. The system then searches for the text pieces whose number sequences are most similar to the question. The advantage over classic keyword search is obvious. Someone asking about “payroll statement” will also find paragraphs about “pay slip,” even though no word matches.

In practice, both methods are combined, since keyword search remains unbeatable for product numbers or proper names. In the end, typically three to ten text pieces, together with the original question, go to the language model. A common misconception is that more hits automatically deliver better answers. Too much material dilutes the answer, because the model also reads along the ill-fitting passages.

Retrieval in chatbots, search, and corporate knowledge

The process is most commonly encountered as RAG, short for Retrieval Augmented Generation. That is the combination of retrieval and subsequent text generation. When a chatbot places links to websites underneath its answer, it has performed a retrieval beforehand. The same thing happens with the summarizing text boxes at the top of search engines.

The approach is especially widespread in companies. An insurance company lets its employees query contract terms in plain language instead of leafing through thousands of PDF pages. Banks and law firms use the same technique for rulebooks. The crucial point: the internal documents remain in the company’s own database and don’t need to be trained into a model.

In business news, retrieval therefore often comes up in connection with vector databases, the specialized databases for such number sequences. Providers in this space have raised large sums of money in recent years. Anyone trying to make sense of such reports should keep one thing in mind: retrieval doesn’t make a model smarter, only better informed.