Tau Scaling

A language model does not write text all at once, but word-fragment by word-fragment. Before each step, it calculates a probability for thousands of possible continuations. These probabilities are not yet the answer, just a suggestion: one word gets 60 percent, another 20, the rest is spread across the remainder. Tau scaling is the dial used to distort this distribution once more before the selection is made. The name comes from the Greek letter tau, which appears in the formulas for this dial. Colloquially, the same dial is often simply called temperature.

The dial between boredom and nonsense

Without this dial, there would only be two unsatisfying options. Either the model always picks the most probable word, in which case it sounds stiff and repeats itself. Or it rolls the dice freely according to the raw probabilities, in which case implausible words regularly slip through. Tau scaling turns this either-or question into a smooth, continuous transition.

This is practically important because different tasks demand different things. Anyone who has a model solve math problems or write program code wants reliability. There, the value is set very low, often between 0.0 and 0.2. Anyone who wants to generate advertising copy, name ideas, or stories sets it higher, typically 0.8 to 1.2.

A common misconception is that a high value makes the model more creative in the sense of smarter. That is not true. The dial changes no knowledge and no ability. It only changes how far the model is allowed to deviate from its own favorite answer. At very high values, above all, the number of freely invented claims increases.

What happens to the probabilities

Technically, the model’s raw values are divided by tau before the final computation step. If tau is smaller than one, the gaps between the candidates become larger. The favorite pulls ahead, while all others fall back. If tau is larger than one, the gaps shrink, and outsiders get a real chance.

An analogy: imagine a class voting on a school trip destination. With low tau, only which destination is in the lead counts, the rest is ignored. With high tau, every vote is treated almost equally, so that even a suggestion with three votes can win. The outcome of the vote changes even though no one changed their mind.

Tau scaling is rarely used on its own. It is usually combined with methods that first filter out the worst candidates entirely. Well-known ones are top-k, which only allows the k best words, and top-p, which only allows as many words as needed to reach a certain total probability. Only after that does the tau dial act on the remainder.

Where the dial shows up in practice

In the interfaces of the major providers, the value is a perfectly normal parameter. Anyone talking to a model via a programming interface usually sets a field there called temperature. In chat interfaces for end users, on the other hand, the dial is often hidden or fixed, so that no one accidentally produces unusable outputs.

The term also explains an observation that irritates many users. The same question yields two different answers on two different days. This is not due to an update, but to the fact that the value is above zero and is re-rolled with each request. Only when tau equals zero is the output largely reproducible.

In technical reports, one also encounters the term in connection with evaluations. When comparing two models in a test, this value must be set identically. Otherwise, one is not measuring the quality of the models, but only the different setting of the dial. This is exactly what is regularly disputed regarding published leaderboards.