Tensor Processing Unit

The main chip in a laptop is an all-rounder. It can display text, play music, and launch programs, but it isn’t particularly fast at any single task. A Tensor Processing Unit takes the opposite approach: it handles almost only one type of calculation, but does so in gigantic quantities. This involves multiplying and adding numbers arranged in large tables with rows and columns. In mathematics, such number tables are called tensors — hence the name. Google began developing these chips itself in 2015 and continues to build them into its own data centers to this day.

Why Google builds its own chips

Every modern AI system is, at its core, made up of huge tables of numbers. When you ask a chatbot a question, your words are converted into numbers and then run through billions of multiplications. A regular main processor would take an extremely long time to do this. A specialized chip that does nothing else is many times faster for the same power consumption.

The second reason is economic. The vast majority of all AI chips worldwide currently come from a single manufacturer, namely Nvidia. Anyone buying from them pays high prices and often waits months for delivery. Google avoids this by having its own chips designed and manufactured at contract factories. This lowers the cost per computing step and makes the company more independent.

The third reason is power. In large data centers, energy is a major cost factor, not just the purchase price of the hardware. A chip that performs the same calculation with fewer watts saves enormous sums over the years. This is why TPUs regularly come up in discussions about the energy appetite of AI.

The compute units inside

The heart of a TPU is a grid of very many small computing units. Each one can only multiply two numbers and add the result to a running sum. The numbers are pushed through this grid like workpieces on an assembly line. Each unit grabs its value as it passes by and hands it on. This means the slow memory doesn’t have to be accessed constantly, which is normally the biggest bottleneck.

A second trick is the precision of the numbers. A regular processor usually calculates very precisely, with many digits after the decimal point. AI models barely need this precision: whether a value is 0.7182 or 0.72 almost never changes the result. TPUs therefore deliberately calculate more coarsely. This allows more computing units to fit on the same chip, and each calculation runs faster.

Even so, a single TPU is still too weak for large models. Google therefore connects thousands of chips via its own high-speed links into what’s called a pod. This cluster works on a shared training run and behaves, from the software’s point of view, like a single very large computer. Incidentally, a common misconception is that TPUs are simply faster graphics cards. Graphics cards are more flexible and usable for many tasks, while TPUs are more tailored to AI and, as a result, less versatile.

TPUs in products and stock market news

You use TPUs regularly without noticing. Google Search, the translation feature, face recognition in Google Photos, and the AI models of the Gemini series largely run on this hardware. The chips can’t be bought. Google rents them out by the hour through its cloud platform, so that research groups and companies can also compute on them.

In business news, TPUs usually appear in connection with Nvidia. Every announcement about a new TPU generation or about a major customer switching from Nvidia to Google moves stock prices. The underlying question is whether Nvidia’s current market dominance in AI chips is permanent. Amazon, Microsoft, and Meta are also now developing their own AI chips for the same reasons.

For everyday use on a phone, there’s a small relative: many smartphones contain tiny AI computing units that enhance photos or recognize speech without sending data to the internet. The principle is the same, just on the scale of milliwatts instead of megawatts.