Tensor Network

A tensor is basically a multidimensional table of numbers. A normal table has rows and columns — that is, two dimensions. A tensor can have three, four, or even more dimensions. That sounds abstract, but it’s actually everyday: a color image is a tensor, because every pixel has three values (red, green, blue) as well as a position in width and height. A tensor network arises when many such tables are linked together instead of using a single giant table. The goal is efficiency: breaking things down into many small pieces saves considerable memory and computing time.

Why the size of tensors becomes a problem

Modern AI models work with tables that quickly reach astronomical sizes. If you double the number of dimensions of a tensor, the amount of data doesn’t grow linearly but explodes exponentially. This growth is called the curse of dimensionality. A tensor with just 100 entries per dimension and 10 dimensions would have more entries than there are atoms in the visible universe. No computer in the world can store something like that directly.

This is exactly where tensor networks come in. They get around the problem by never writing out the huge tensor in full in the first place. Instead, they describe it as the interplay of many small tables. The information is then not contained in one gigantic structure, but distributed across the connections between the small pieces. This works well because most tensors in practice aren’t actually random — they contain patterns and redundancies that can be compressed.

How the decomposition of a tensor works

Imagine describing a long piece of music. Instead of writing down every single note, you could record chords, rhythm, and melody separately and explain how they fit together. The result is the same piece of music — but the description is much more compact. Tensor networks work on the same principle. The original tensor is split into several smaller tensors. These small tensors are connected to one another via so-called indices: shared dimensions through which the pieces interact.

A well-known form is matrix decomposition, in which a large two-dimensional table is split into two narrow tables that are multiplied together. Tensor networks generalize this to any number of dimensions and any number of sub-pieces. Depending on how the pieces are connected to one another — in a chain, a tree, or a mesh — a different type of tensor network emerges, each with its own strengths.

An important concept here is rank. Rank describes how much information is allowed to flow through a connection between two sub-pieces. A low rank means more aggressive compression — faster, but less accurate. A high rank is more precise, but more expensive. Researchers choose the rank depending on the task at hand.

Tensor networks in AI models and quantum computing

In AI, tensor networks mainly show up wherever models need to be compressed or accelerated. One concrete application is the compression of weight matrices — that is, the learned number tables — in large language models. Instead of keeping a huge matrix in memory, you store its tensor network representation. The model remains almost as capable, but requires significantly fewer resources. This is crucial for deploying AI on smartphones or in cars.

Even older is the connection to quantum physics. Quantum states — the states of tiny particles — can be represented as tensors, and simulating them is barely possible classically. Since the 1990s, tensor networks have been the most important tool in theoretical physics for computing quantum systems despite this difficulty. Quantum computing companies like Google or IBM use tensor network methods to simulate and test their machines.

In the news, the term often comes up in connection with “efficient AI” or “model compression.” If you read that a model was shrunk by a factor of four through decomposition methods, you’re often dealing with some variant of tensor networks — even if that name isn’t always explicitly mentioned.