Softmax Function

An AI system internally computes with perfectly ordinary numbers. At the end of such a computation stands, for example, a list like 2.1 / 0.4 / -1.3 — one number for each possible answer. Initially, these numbers only say: the bigger, the more likely. What’s missing is a statement about how confident the system is. That’s exactly what the softmax function delivers: it converts the list into percentage values that together add up to exactly 100 percent. From 2.1 / 0.4 / -1.3 you then get something like 82 percent / 15 percent / 3 percent.

Why numbers first have to become percentages

Raw numbers can be compared but not interpreted. Is a 2.1 a confident result or a close one? That depends on how high the other numbers are. Percentage values solve this problem because they are always relative to one another. Only then can you say: the model is 82 percent confident — or just 34 percent.

This information is practically valuable. A medical image-analysis system that detects an anomaly with only 40 percent confidence should bring in a human. At 98 percent, that’s less urgent. Without softmax, this gradation wouldn’t exist — only a raw ranking of the possibilities.

This step is also necessary for learning. A model is trained by measuring its error and slightly adjusting its internal values. This error measurement requires the output to be a probability. The predicted percentages are then compared with what was actually correct. Without softmax, there’s no yardstick for the comparison.

Exponents, sum, share

The calculation consists of two steps. First, each number in the list becomes the exponent of the number e, roughly 2.718. So 2.1 becomes about 8.2, and -1.3 becomes about 0.27. This step has two effects: all values become positive, and large values grow noticeably more than small ones. Afterward, each value is divided by the sum of all values. The result is shares that inevitably add up to 1.

The name is explained by this behavior. A hard maximum selection would take only the largest value and set all others to zero. Softmax makes this softer: the largest value gets the lion’s share, but the others retain a remainder. Hence “soft” instead of hard.

A dial called temperature controls how soft the result turns out. If you divide all the numbers by a value greater than 1 before the calculation, the percentages move closer together and the output becomes more random. At a value below 1, they spread apart, and the favorite dominates almost completely. This very dial is behind the “creative” or “precise” setting in many chatbots.

Softmax in chatbots and image recognition

Every language model uses softmax for every single word it writes. The model evaluates tens of thousands of possible next word fragments, softmax turns this into a probability distribution, and then a fragment is drawn from it. Because the drawing is random, you don’t always get the same answer to the same question.

With image recognition, the effect is more visible. A photo app that writes “Golden Retriever, 94 percent” is directly showing a softmax result. Spam filters and translation systems also output their assessments this way.

One common misconception is important to know: the percentage figure is not a statement about truth. A model can be 99 percent confident and still be wrong. Softmax only measures how strongly the model favors one option over the others. Experts call such exaggerated confidence overconfidence, and it is a well-known problem in modern networks.