Activation (neural network)

An artificial neural network is a computational program made up of many small, similar building blocks. Each of these building blocks receives numbers from other building blocks, processes them, and passes on a new number. This very output number is called an activation. If it is large, the building block has found something in its input that it reacts to. If it is zero or close to zero, it has not found anything relevant and stays quiet. The name comes from biology: a nerve cell either fires strongly or not at all.

What activations reveal about the intermediate state

A network does not jump straight from input to answer. It computes in layers, that is, in successive stages. After each stage there is an intermediate result, and this intermediate result consists of nothing other than activations. You can imagine them as notes that the network jots down along the way.

For researchers, these notes are the only view into the inside. In an image recognizer, one can measure which units show high values for pictures of cats. In language models, researchers look for activation patterns associated with, say, politeness or a particular language. This field of research is called interpretability, and it aims to explain why a model answers the way it does.

Activations also cost memory, and a lot of it. During training, the system must retain the intermediate values in order to trace them back later. In large models, these values fill up graphics card memory faster than the model itself does. That’s why the term is not merely theoretical, but a hard technical limit.

From the sum to the number: the activation function

A building block computes in two steps. First, it forms a weighted sum: each incoming number is multiplied by a weight, then everything is added together. Weights are the values that the network adjusts during learning. This sum is not yet the activation.

In the second step, the sum passes through what is called an activation function. It is a small computational rule that bends the result. The most common one is called ReLU and is remarkably simple: negative values become zero, positive values remain unchanged. Other variants squeeze everything into the range between zero and one.

This bending is not a mere detail, it is the crux of the matter. Without an activation function, all layers would be nothing but additions and multiplications. A network with a hundred layers could then do no more than a network with a single one. Only the nonlinear kink makes it possible to learn complicated relationships. A common misconception is to equate activation with activation function: the function is the rule, the activation is its result.

Where the term appears in news and tools

Anyone reading about AI usually encounters activations in the context of safety and control. Companies such as Anthropic or OpenAI publish work in which they search the model for units that fire on dangerous topics. Sometimes they deliberately amplify or dampen such values in order to change the behavior. This is called activation steering.

The term also lurks, often hidden, in product announcements. When a provider advertises longer context windows, that is, more text per request, what’s going on in the background is memory for intermediate values. Techniques such as activation checkpointing discard intermediate results and recompute them later in order to save memory.

For you personally, the term is above all a tool for understanding. When you hear that a model has billions of parameters, those are the weights, that is, the stored knowledge. Activations, by contrast, are freshly generated with every single request and disappear again afterward. Weights are the memory, activations are the current thought.