Candidate Generation

A video platform might offer a billion videos. On its homepage, it shows twenty of them. It’s impossible to find these twenty by carefully evaluating every single video. The computing time required would be far too long, and the user only waits milliseconds. That’s why such systems work in two stages. The first stage is called candidate generation: it roughly and very quickly pulls a few hundred entries out of the huge pool that could possibly be relevant at all. This preliminary selection is called the candidates.

Why nobody evaluates a billion videos one by one

The reason is pure computational arithmetic. A precise scoring model might need a millionth of a second per entry. For a billion entries, that adds up to about a thousand seconds, or more than a quarter of an hour. For a single homepage. That’s not just too slow, it would also be unaffordable, since every request costs electricity and server time.

The two-stage split solves the problem elegantly. Candidate generation is allowed to be imprecise, but it has to work extremely fast. The second step, so-called ranking, is allowed to be slow and elaborate, but only has to sort a few hundred candidates. Together, this yields response times under a tenth of a second.

One detail here is important and often overlooked: whatever the first stage discards is gone for good. The best video in the world is useless if it doesn’t make it into the candidates. That’s why this stage isn’t judged on whether it gets the order right, but on whether the good matches are included at all. Experts call this recall.

From the search query to a few hundred matches

A common method works with vectors. Each entry is translated into a long list of numbers describing its content. Two similar videos get similar number lists. The user, too, gets such a list, calculated from their past behavior. Candidate generation then means: search for the entries whose number lists are closest to the user’s.

To make this fast, the vectors are stored in advance in a special data structure, a vector index. It works similarly to the index at the back of a book. You don’t have to read every page to find a passage. Such indexes don’t guarantee delivering the exact nearest neighbors, only with very high probability – but in exchange, they’re a thousand times faster.

In practice, several generators usually run in parallel. One searches for topical similarity, one for popularity, one for novelty, one for what friends have watched. Each delivers its own list, and all the lists are thrown together. This creates a mix that doesn’t get stuck in a single line of thinking.

Streaming services, online shops, and AI chatbots with source access

Every recommendation strip you know relies on this principle. The suggestions on YouTube, the Netflix homepage, the “You might also like” row in an online shop. Classic search engines work this way too: first documents with matching words are collected, then they’re sorted.

The term comes up particularly often these days in connection with AI chatbots that access company documents. This method is called RAG, short for Retrieval Augmented Generation. Before the language model answers, a candidate generator retrieves the presumably relevant text passages. The model only gets to read these few passages. If the preliminary selection is poor, even the best model can’t help anymore.

In corporate announcements, you can often spot this topic through phrases like “retrieval pipeline” or “vector database.” Behind them lies almost always the question of how a company quickly pulls the right few items out of very large amounts of data. That is exactly candidate generation.