Language Processing Unit

Programs like ChatGPT write their answers word by word. Every single word costs computational effort, and this work has so far mostly been handled by graphics cards. A Language Processing Unit is a computer chip built for exactly this one task: generating text. It is meant to accomplish this faster and with less power than a graphics card, which was originally developed for images and games. The term became known through the US company Groq, which coined the name for its own chips. Other manufacturers call similar specialized chips by different names, such as inference accelerators.

Why speed in answering is worth money

Anyone operating a language model pays for every single answer. The cost doesn’t arise during the model’s training, but during day-to-day operation. Experts call this operation inference: the finished model is being used, not trained anymore. Millions of requests per day add up to considerable electricity costs and hardware costs.

Then there’s the impression on the user. A chatbot that spits out words at a rate of one per second feels sluggish. If the words appear faster than one can read, the conversation feels fluid. LPUs advertise several hundred words per second for medium-sized models, where graphics cards are often significantly slower.

The comparison with graphics card market leader Nvidia is important. Its chips can do almost everything: training, generating images, writing text. LPUs forgo this versatility in order to win in their one discipline instead. Whether this pays off in the long run is one of the more exciting bets in the chip industry.

The assembly-line trick inside the chip

The bottleneck in text generation is rarely the computation itself. It is the waiting for numbers from memory. A language model consists of billions of stored numerical values, the parameters. For every new word, many of these must be retrieved. A graphics card fetches them from a large external memory chip, and this path costs time.

LPUs take a different approach. They pack a very large amount of fast memory directly onto the chip, right next to the computing units. The path becomes shorter, the wait becomes shorter. The price for this: the memory on the chip is small. A large model doesn’t fit on a single LPU chip; many of them must be connected together.

On top of that comes strict scheduling. With a graphics card, the hardware decides during operation which task runs when. With an LPU, software determines this in advance, cycle by cycle. You can imagine it like an assembly line whose schedule is fixed down to the second. This runs very smoothly and without idle time, but hardly tolerates any deviations from the plan.

LPUs in cloud services and stock market news

As a private individual, you will hardly ever buy an LPU directly. You encounter it via the internet: providers place their chips in data centers and sell access to them. Developers send their requests there and get answers back without ever seeing the hardware. Some chat services and voice assistants therefore seem noticeably fast.

In business news, LPUs mainly appear as competition to Nvidia. Companies like Groq or Cerebras are raising billions of dollars from investors, promising to offer cheaper inference. Google, Amazon, and Microsoft are also building their own specialized chips, but use different names for them.

A common misconception: LPU is not a protected category like CPU or GPU. It is a marketing term coined by a single manufacturer that has become widespread. Anyone reading about LPUs in a news report should therefore check exactly what figures were measured. Speed claims often apply only to a specific model and a specific request length.