Sigmoid

The sigmoid function is a mathematical curve with a characteristic S-shape. It takes any number — whether 1000 or −1000 — and always returns a value between 0 and 1. Very large positive numbers end up close to 1, very large negative numbers close to 0, and the number 0 lands exactly at 0.5. This property makes the sigmoid function useful whenever you want to express a statement as a probability: a value of 0.9, for example, can be read as “90% certain.”

Sigmoid as a switching point in the neural network

Neural networks — roughly speaking, systems made up of many interconnected computing units — process data in layers. Each unit sums its inputs and passes the result on to the next layer. The problem: without restriction, these numerical values grow without bound. The sigmoid function acts like a valve that forces the output into the range 0–1 after each unit.

This step is called activation. Without such an activation function, the entire network would mathematically be nothing more than a single linear calculation — no matter how many layers you stack. The sigmoid function breaks up this linearity and allows the network to learn more complex relationships.

How the S-curve arises

The formula is σ(x) = 1 / (1 + e^(−x)), where e is Euler’s number (approximately 2.718). Anyone who doesn’t know this formula doesn’t need to memorize it. The crucial property lies in the structure: the denominator is always at least 1, because e^(−x) never becomes negative. This means the result is always less than 1. And because the numerator is fixed at 1, the result is always greater than 0.

The flat region on the left and right edges of the curve is also its biggest weakness. There, the output barely changes anymore, no matter how much the input changes. During the training of a network — that is, the process in which the network adjusts its parameters — the slope of the curve is needed as a signal. In the flat region, this signal is close to zero. The network then learns almost nothing at these points. This phenomenon is called the vanishing gradient.

Because of this problem, the sigmoid function was largely replaced in the middle layers of deep networks by other activation functions, above all by ReLU (Rectified Linear Unit), which simply sets all negative values to zero and otherwise remains linear. However, sigmoid remained in use as an output function wherever a probability is required as the result.

Where sigmoid still appears today

In binary classification — that is, tasks with exactly two possible answers, such as “spam or not spam?” — sigmoid remains the standard choice in the output layer to this day. The value 0.5 serves as the threshold: if the output is above it, the model decides for one class, below it for the other.

The sigmoid function is also embedded in logistic regression — a classic statistical method that predates modern AI. And in language models, i.e. systems such as chatbots, it appears in certain control mechanisms, for example in so-called gating: here, a sigmoid value between 0 and 1 decides how much information from a previous computation is allowed to flow into the current one.

So anyone reading reports about new AI models and coming across terms like “activation function,” “output probability,” or “gating” is moving in the direct vicinity of the sigmoid function — even if the word itself isn’t always mentioned.