Rejection Sampling

Rejection Sampling translates roughly to “sampling with rejection”. The principle is simple: many random proposals are generated, and only those that pass a check are kept. All others are thrown away, without being fixed. One thus accepts wasting a lot of effort, because checking is easier than deliberately producing a good result. A program that writes texts can, for instance, formulate ten answers and output only the one that best passes an evaluation. The idea originally comes from statistics, but today it is encountered above all in the training and operation of language models.

Why throwing away is often cheaper than thinking harder

A program that generates text makes a choice with an element of randomness for every word. That’s why the same question never produces exactly the same answer. Some of these answers are good, some contain errors or fabrications. There is no simple way to force the program to hit the best version immediately.

Checking, on the other hand, is often easy. Whether a piece of program code runs can be determined by a computer in seconds. Whether a math problem has been solved correctly is revealed by a glance at the result. This gap between “hard to produce” and “easy to check” is the actual reason why the method pays off.

For companies building AI systems, this is a very practical lever. They don’t need a new model to get better answers. It’s enough to put more computing time into multiple attempts. The price for this is honestly visible: with ten attempts, an answer costs roughly ten times as much.

Generate, evaluate, discard

The process has three steps. First, the model generates several answers to the same request, often between four and a hundred. Then a checker evaluates each individual answer. Finally, only the best one remains, and the rest is discarded.

The checker can take very different forms. For programming tasks, it’s a test run of the code. For mathematics, it’s a comparison with the known solution. For open-ended texts, a second model often takes over the evaluation, a so-called reward model. This model was previously trained on human judgments to distinguish helpful answers from bad ones.

The second use is particularly important: the good answers are not only kept for the user, but are also collected as training material. The model is subsequently retrained on its own best outputs. This procedure is called rejection sampling fine-tuning and was, for example, a building block in Meta's Llama models. A common misconception is that the method directly makes the model smarter. It only filters – the quality can never be better than the best answer the model would have produced anyway.

Where the discarded answers hide in everyday life

If a chatbot conspicuously takes a long time on a difficult math problem, Rejection Sampling may be behind it. Several solution paths run in parallel in the background, but only one becomes visible. Programming assistants work similarly, generating several code suggestions and automatically running them against tests. Image generators also discard results, for instance when a filter detects prohibited content.

In trade news, the term usually appears in two contexts. First, in connection with “test-time compute”: this refers to a system using more computing power when answering rather than during training. Second, in comparison with reinforcement learning, learning through reward. There, feedback continuously changes the model’s internal values, whereas Rejection Sampling only selects and is therefore simpler and more stable.

For you as a user, this means above all one thing: answer quality is partly a matter of budget. Free access often provides only one attempt, paid modes several. The difference then lies not in the model’s knowledge, but in the number of discarded drafts.