MLP Layer
An MLP layer is a fundamental building block in modern AI models that processes and transforms information through simple computational operations. It is present in almost every language model and is largely responsible for how much a model "knows".
A neural network — that is, an AI model loosely modeled on the structure of the brain — consists of many layers that process data step by step. An MLP layer (short for Multilayer Perceptron, roughly “multi-layer network of computational nodes”) is one of these layers. It takes in a list of numbers, performs calculations with it, and passes on a new list of numbers. That sounds abstract, but it is the core of almost everything modern AI models like language assistants accomplish. Concretely: each MLP layer consists of two successive computational steps — a multiplication and a simple filter function that filters out or distorts negative values. This combination repeats dozens to hundreds of times throughout the model.
Why MLP layers carry so much weight
In modern language models, two types of layers alternate: attention layers and MLP layers. Attention layers decide which words or concepts in the text relate to one another. MLP layers, on the other hand, further process these relationships and, in doing so, store factual knowledge.
Researchers have found that MLP layers contain a large portion of a model’s stored factual knowledge — things like “Paris is the capital of France” or “water consists of hydrogen and oxygen”. A language model without MLP layers would be good at recognizing sentence structures but would have hardly any content to reproduce. MLP layers together usually account for more than half of the total model size.
Calculating in two steps
The process within an MLP layer is always the same. First, the incoming list of numbers is multiplied by a so-called weight matrix — a large table of learned numbers. This produces a new, often much longer list of numbers. This step “stretches” the information into a larger space.
Next comes an activation function. It is comparatively simple: it lets high values pass through unchanged and suppresses or alters low values. This filter ensures that not all computational nodes “fire” at the same time — only the relevant ones remain active. The list of numbers is then compressed back to its original length before being passed on to the next layer.
A common misconception: many people think the MLP layer “thinks” or “understands”. It does neither. It performs only matrix multiplications and filter functions — these are simple mathematical operations. The apparent intelligence only emerges because millions of such operations are carried out one after another.
MLP layers in products and headlines
Every time someone uses ChatGPT, Gemini, or a similar language assistant, hundreds of MLP layers run in the background. Models like GPT-4 or LLaMA 3 often have more than 80 such layer pairs of attention and MLP. The sheer number of these layers is one of the reasons why these models require so much computing power and memory.
In research, the term frequently appears in connection with the so-called Mixture-of-Experts method. In this approach, the MLP layer is split up: instead of a single large layer, there are many smaller “expert MLP layers”, only a few of which are active per request. This saves computing time without reducing the model’s knowledge.
MLP layers also play a central role when it comes to model size. When headlines state that a model has 70 billion “parameters” — that is, stored numerical values — most of them reside in the weight matrices of the MLP layers. Anyone who understands what an MLP layer is also understands why larger models don’t simply become faster, but above all more memory-hungry.
