Transformer Architecture

When a computer program is meant to process language, it needs a fixed blueprint for how it takes in words, computes with them, and passes them on. The transformer architecture is exactly such a blueprint. It was introduced in 2017 by researchers at Google in a paper titled “Attention Is All You Need.” Its central idea: while processing a sentence, the program looks at all the words simultaneously and calculates which words are important to one another. Older approaches instead read texts word by word from left to right, like reading aloud. The transformer discards this ordering and thereby gains enormously in speed.

Why 2017 became a turning point

Before the transformer, language programs were slow to train. They had to go through a sentence step by step, because each word built on the result of the previous one. This made it nearly impossible to distribute the work across many computing chips. The transformer processes all words in parallel, that is, simultaneously. This fits perfectly with graphics cards, which can carry out thousands of small calculations at once.

This property had a consequence that almost no one foresaw at the time. One could simply keep making the models bigger and feed them ever more text. And with every increase in size, they became noticeably better. The entire boom of recent years arose from this observation.

Today, this architecture is embedded in practically everything sold as “AI.” The “T” in GPT stands for Transformer. Image generators, translation services, and programs for analyzing proteins now also use the same basic structure. So a single blueprint serves very different fields of application.

Attention: the core of the blueprint

First, the text is broken down into small units called tokens. A token is usually a word or part of a word. Each token is assigned a long list of numbers that describes its meaning. Similar words receive similar number lists. Additionally, the position of the token within the sentence is recorded, since otherwise the order would be lost.

Then comes the actual trick, the attention mechanism. For each token, the model calculates how strongly it should attend to every other token. In the sentence “The cat chased the mouse because it was hungry,” the word “it” must refer to “cat,” not “mouse.” The mechanism assigns a high weight to “cat” here. This is how individual words give rise to an understanding of context.

This step runs multiple times in parallel, which is called multi-head attention. One pass might focus on grammar, another on meaning. This is followed by a regular computational layer that further processes the result. This double step is repeated dozens of times in succession. Large models stack up to a hundred such layers on top of one another.

Where transformers show up in everyday life

Every chatbot you use works with this architecture. The same goes for translation tools like DeepL, autocomplete in search engines, and coding assistants like GitHub Copilot. Automatic video captioning is also based on it. The term itself usually remains invisible, because products are marketed under brand names rather than blueprints.

In business news, you encounter the transformer indirectly. Because the architecture runs so well on graphics cards, the chipmaker Nvidia has become one of the most valuable companies in the world. Reports about new data centers and their power consumption are also directly tied to this technology.

A common misconception is that transformer and language model are the same thing. The transformer is only the blueprint; the model is the finished, trained result. The same blueprint can be used for text, images, or music. A well-known weak point remains the computational cost: if you double the text length, the work in the attention step quadruples. Research is currently working intensively on exactly this problem.