Inference chip

An AI system like a chatbot goes through two very different phases. First, it learns from huge amounts of data, which can take weeks and happens only once. After that, it gets used: someone types in a question, the system computes, an answer appears. This second phase is called inference, meaning the application of the finished model. An inference chip is a computer chip built exactly for this second phase. Its job is to compute a single answer as quickly and with as little power as possible.

Why the real cost only shows up during operation

Training a large model is a one-time investment. Applying it, by contrast, runs continuously, every day, for every single request. If a service has a hundred million users, costs arise anew with every answer. That is exactly why, at many companies, application costs now make up the larger share of the bill.

A second point is power consumption. Data centers for AI use as much energy as small cities. A chip that needs thirty percent less power per answer saves millions and eases the strain on the grid. That’s why manufacturers today measure their chips not just in raw computing power, but in performance per watt.

Then there’s latency. During application, a human is sitting there waiting for the answer. A two-second delay is immediately noticeable, whereas during training a one-second delay wouldn’t matter at all. That’s why inference chips are optimized for short response times, not just high throughput.

What sets these chips apart from training hardware

At their core, AI models always compute the same thing: a huge number of multiplications and additions of numbers arranged in large tables. An inference chip contains thousands of simple computing units that carry out this one task in parallel. Everything else a normal processor can do is deliberately left out. The comparison fits a workshop: a general-purpose processor is the Swiss Army knife, an inference chip is the specialized machine that performs only one motion, but does it millions of times over.

The second major difference lies in numerical precision. During training, a model needs fine-grained intermediate steps, so calculations there use long, precise numbers. During application, coarser numbers are almost always sufficient. This is called quantization: instead of storing sixteen digits, the chip stores only eight or even four. This halves or quarters the memory requirement and speeds up computation considerably.

A common misconception is that raw computing power is what decides performance. In reality, memory is usually the bottleneck. The model’s values constantly have to travel from memory to the computing units. That’s why manufacturers now build memory directly next to or on top of the chip, so the distances stay short.

From the data center to the smartphone

The best-known examples sit in data centers. Google has been building its own chips called TPUs for years, and Amazon uses a line called Inferentia. Nvidia dominates the market for training hardware, but it also sells chips designed primarily for application. Startups like Groq or Cerebras are attacking exactly this market, because it’s where the bigger growth is expected.

The second location is the device itself. Almost every newer smartphone contains a small unit for AI computations, often called an NPU. It recognizes faces in the photo gallery, translates text without an internet connection, or filters background noise during video calls. That is inference in miniature, right in your pocket.

That’s why inference chips regularly turn up in business news. When a corporation announces it is developing its own chips, the goal is usually to become less dependent on Nvidia and to cut ongoing costs. For investors, the interesting part is that the market for application is likely to be larger in the long run than the market for training.