Sampler

A program like ChatGPT writes its answers piece by piece. After each chunk of text, it calculates how likely every possible next chunk is. The result is not a single decision, but a long list of probabilities. For “The capital of France is,” “Paris” might get 95 percent, “Lyon” one percent, and thousands of other words almost nothing each. The sampler is the part of the program that actually picks one from this list. It doesn’t think anything up, it rolls the dice according to the rules it’s given.

Why the same question gets two different answers

Many people believe an AI model has one fixed answer to every question. That’s not true. If you ask the same question twice, you often get different phrasings. The reason is almost never the model itself, but the sampler. It draws anew from the probability list each time, and the choice doesn’t always come out the same.

This randomness is intentional. A sampler that always stubbornly picks the most likely word produces surprisingly bad text. It repeats itself, goes in circles, and sounds stilted. A bit of randomness brings variety to the sentences. Too much randomness, on the other hand, causes the model to write nonsense or switch topics mid-sentence.

For companies, this is a practical problem. Anyone using AI for legal texts or program code wants as little surprise as possible. Anyone generating advertising copy or stories wants exactly the opposite. That’s why almost all providers offer settings that let developers control the sampler.

Temperature, Top-k, and Top-p

The best-known dial is called temperature. It changes how much the differences between probabilities matter. A low temperature, say 0.2, sharpens the differences: the favorite is chosen almost every time. A high temperature, say 1.2, flattens the list and gives even unlikely words a real chance. At temperature zero, randomness disappears entirely, and the model always picks the top candidate.

Two other methods trim the list beforehand. Top-k keeps only the k best candidates, for example the 40 most likely, and discards the rest. Top-p, also called nucleus sampling, works more flexibly: it gathers candidates until their probabilities together reach a threshold, often 90 percent. If the model is very confident, only a few words remain. If it’s uncertain, more come into play.

In practice, these dials are combined. First, top-p cuts away the absurd candidates, then temperature determines how boldly the dice are rolled within the rest. Think of it like a restaurant menu: top-p decides which dishes are even on the menu, temperature decides how adventurous you are when ordering. It’s important to note the difference from training: the sampler does not change the model. It’s a setting used at inference time and can be changed at any moment.

Samplers in chatbots and image generators

Anyone using an AI only through an app or website usually doesn’t see these settings. The provider has set them in the background, often to a middle value that works well for everyday questions. They become visible as soon as you address a model via a programming interface. There, temperature and top-p appear as perfectly normal parameters that you send along with each request.

Image generators like Stable Diffusion also talk about samplers. There the word means something different: procedures with names like Euler or DPM++ control the steps by which a finished image emerges from image noise. Different samplers produce slightly different images given the same input and take different amounts of time. What both meanings have in common is the basic idea: a procedure draws, step by step, one concrete outcome from many possibilities.

In reviews and expert discussions, the term often comes up as an explanation for odd behavior. When a chatbot suddenly writes gibberish, that’s sometimes due to too high a temperature and not to a weak model. Conversely, answers at temperature zero seem reliable but boring and repetitive. Anyone who knows the dials can make much better sense of such observations.