MLP Cross Layers

An AI model consists of many computational layers arranged one after another. Each layer receives numbers from the layer before it, transforms them, and passes them along. A common building block within such a layer is a small network of computational nodes that combines every number with every other number. Such building blocks are called MLPs, short for Multi-Layer Perceptron. With MLP Cross Layers, this building block is redesigned so that it doesn’t only communicate with its own layer, but also has connections running across to other layers. That’s exactly what the word “cross” means: crosswise, extending beyond the boundaries of an individual layer.

What cross-connections are meant to save

Large language models are expensive because they contain an enormous number of adjustable numbers. These numbers are called parameters, and the largest share of them sits precisely in the MLP building blocks. In some models, two-thirds of all parameters are accounted for by them. Every parameter takes up memory on the graphics cards and costs computing time on every request.

If an MLP building block can draw on information from other layers, it has to learn less on its own. Some things that would otherwise be freshly stored in every layer now reside in only one place. The idea, then, is: the same performance with less weight. For providers, that means lower operating costs; for users, faster responses.

It’s important to distinguish this from Mixture of Experts. There, computing time is saved by activating only part of the building blocks per request. Cross Layers pursue a different goal: they change how the building blocks are wired to one another. The two approaches are not mutually exclusive and are sometimes combined.

The wiring between the levels

Normally, a strict order applies. Layer 5 only sees what layer 4 passes on to it. With Cross Layers, there are additional pathways. A building block in layer 5 can also read intermediate results from layer 2 or layer 3. These pathways are built in permanently and are learned along with everything else during training.

One variant goes even further and has several layers share the very same building block. The MLP block then exists only once but is used at multiple points. This is like a tool passed around a workshop instead of buying one for every workstation. Memory requirements drop noticeably, but computing time per pass does not automatically follow suit.

One catch remains. Cross-connections make the process less regular, and graphics cards prefer regularity. If intermediate results from earlier layers have to be kept around, that costs additional memory. Whether it pays off depends heavily on the specific model. That’s why this approach is, for now, more a research topic than a standard.

In research papers and model cards

As a user of a chatbot, you notice nothing of Cross Layers. The term shows up where providers describe the internal architecture of their models. Such descriptions appear in technical papers and in so-called model cards, which are technical fact sheets for a published model.

It comes up especially often with small models meant to run on phones or laptops. There, memory is scarce, and every billion parameters saved counts. Similar ideas circulate under names like Layer Sharing or Cross-Layer Parameter Sharing. The terms are not used consistently, but they usually refer to the same underlying principle.

A common misconception is that such tricks make the model smarter. As a rule, they don’t. They make it smaller or cheaper, while keeping quality as steady as possible. When an announcement talks about a more efficient model, it is almost always referring to exactly this kind of restructuring.