Recurrent Neural Network

Some data has a fixed order. A sentence consists of words that come one after another. A stock price consists of prices measured one after another. A recurrent neural network is a trainable computer program that works through such sequences step by step. After each step, it stores a kind of short note about everything it has seen so far. This note flows back into the next step. This is exactly where the word recurrent comes from: the program draws on its own previous result.

Why order is a problem at all

Older programs of this kind always received a fixed amount of data at once, for example an image with a fixed number of pixels. For language, this works poorly. Sentences are sometimes five words long, sometimes fifty. A recurrent network doesn’t have this problem, because it simply keeps running for as long as there is data.

On top of that comes the importance of position. The sentences “The dog bites the man” and “The man bites the dog” contain the same words. Their meaning is nevertheless completely different. A model that treats words merely as an unordered pile cannot capture this difference. Recurrent networks were the first widely usable approach that took order seriously.

That is why, from around 2014 onward, they were built into almost every language product. Machine translation, speech recognition on phones, and the word suggestions on a phone keyboard all ran on this design. Even today, they are still in use in many older systems that nobody rebuilds without good reason.

The loop and its memory problem

The core is a loop. The network takes the first word and computes a state from it, that is, a list of numbers. For the second word, it computes again, but this time uses the word and the old state together. This produces a new state, and this repeats until the end of the sentence. You can imagine this like someone who, while listening, constantly overwrites their note instead of keeping each word separately.

This note has a fixed size. A long text must therefore fit into the same list of numbers as a short one. In practice, each new step overwrites part of the old content. After forty or fifty words, hardly anything remains of the beginning. Experts call this disappearance of information the vanishing gradient problem.

Improved variants were developed to counter this forgetting, above all LSTM and GRU. Both build in small switches that decide which information is kept and which is deleted. This helped considerably, but did not fully solve the problem. A second drawback remained regardless: because each step waits for the previous one, the computation is hard to distribute across many processors.

From standard to footnote in AI news

It was precisely at this point that the Transformer, the architecture behind ChatGPT and similar systems, stepped in in 2017. It looks at all words simultaneously instead of one after another. This can be computed in parallel across thousands of graphics cards and is what makes training huge models possible in the first place. Within a few years, it displaced recurrent networks from almost all major language applications.

Even so, they have not disappeared. For short sequences and limited computing power, they are often the better choice. Sensors in machines, heart rate monitoring in clinics, or small models running directly on a device continue to use them. They require little memory and deliver results immediately as new data arrives.

In the news today, the term mainly appears as a point of comparison. When new architectures such as Mamba or other so-called state-space models are discussed, it is often said that they bring back the idea of recurrence. The reason is economic: Transformers become expensive with very long texts, whereas step-by-step methods do not. A common misconception is therefore to consider recurrent networks definitively outdated.