Gate

Programs that learn from examples instead of following rigid rules consist of many small computational steps. Between these steps, numerical values flow back and forth. A gate is a point at which such a flow of numbers is throttled. Picture a water tap: it lets everything through, nothing at all, or something in between. What’s special is that the position of this tap isn’t fixed in place. It’s recalculated for every input, and the program itself learns when to open it and when to close it. The German term for this would be “Tor” or “Gatter,” but the English word is the one commonly used.

Why networks need a water tap

Without gates, a network treats every piece of information the same way. Everything flows onward with the same strength, regardless of whether it’s important or not. For short tasks, this barely matters. For long texts or long sequences of numbers, it becomes a problem. The model overwrites important information with unimportant information because it has no way to hold onto anything.

A second problem concerns learning itself. Networks learn by having an error signal travel backward through all the computational steps, adjusting the settings along the way. Over many steps, this signal becomes weaker and weaker and eventually nearly vanishes entirely. Experts call this the vanishing gradient problem. Gates create shortcuts along which a value is passed on almost unchanged over many steps. This allows the error signal to still reach points far back in the network.

The third reason is economic. Large models have so many building blocks that it would be too expensive to have all of them compute for every request. Gates can switch off entire parts of the network. This is what keeps a very large model affordable to run in practice.

From sigmoid to gate position

Technically, a gate is usually a number between 0 and 1. This number is multiplied with the incoming signal. At 0, nothing gets through; at 1, everything does; at 0.3, just under a third. To guarantee that the number falls within this range, a squashing function called sigmoid is used. It compresses any given computed value into a value between 0 and 1.

Which value the gate outputs depends on the current input. A small dedicated computation unit looks at what’s currently arriving and derives the gate position from it. The settings of this computation unit are learned along with everything else during training. So no one sets by hand when the gate should open. The model discovers the rule itself from the training data.

Gates are well known from an older network architecture called LSTM, which processes text word by word. It has a forget gate, an input gate, and an output gate. The forget gate decides how much of the existing memory is retained. A common mistake is confusing these gates with the logic gates in computer chips. Those only know on and off, whereas gates in networks know every intermediate stage.

Gates in today’s language models

In reports about new AI models, the word usually comes up in connection with the router. The router is the gate of large models that consist of many specialized building blocks. For each word, it selects which of these building blocks are allowed to respond. The technical English term for this is gating network.

Gates are also embedded inside almost every current language model, without this being widely reported. A common building block called SwiGLU works on exactly this principle: one part of the computation exists solely to dampen or let through another part. Whenever a technical report mentions something being “gated,” this is always the kind of switch being referred to.

For users, this remains invisible, but it directly affects price and speed. A model with good gating responds faster and uses less power because it leaves large parts of itself unused. If the router distributes poorly, some building blocks become overloaded while others sit idle. That’s why the quality of gating is one of the points that development teams write about extensively in their publications.