H100

The H100 is a particularly powerful computer chip made by the US company Nvidia. It wasn’t built for office computers, but for the massive data centers where programs like ChatGPT are created and run. Chips like this are called graphics cards or GPUs because they were originally developed for video games. But their real strength is that they can carry out very many simple calculations at the same time. This is exactly the kind of calculation that modern artificial intelligence needs in enormous quantities. The H100 came onto the market in 2022 and for several years became the most important component of the entire AI industry.

Why a single chip became a political issue

Anyone who wants to train a large language model today doesn’t need one H100, but thousands. A single unit cost around 30,000 dollars at times. A data center with 20,000 such chips therefore costs several hundred million dollars in hardware alone. That’s why the figures Nvidia reports every quarter serve as a fever thermometer for the AI industry among investors.

On top of that, there was hardly any alternative for a long time. Nvidia had a huge lead, not only in hardware but also in the software called CUDA, which programmers use to control the chips. Once you’ve written your programs for CUDA, you don’t easily switch to another manufacturer. This effect is called lock-in and is one reason why Nvidia became one of the most valuable companies in the world at times.

Governments are also interested in the H100. The US has heavily restricted the export of such chips to China, because they can also be used to develop militarily relevant systems. This gave rise to a gray market with significantly inflated prices. A piece of electronics the size of a book cover has thus become a matter of foreign policy.

What’s different about the H100 compared to a gaming graphics card

A normal main processor works through a few tasks one after another very quickly. The H100 does the opposite: it has tens of thousands of small computing units that work in parallel. At their core, AI models consist of matrix multiplications, meaning the mass multiplication and addition of number tables. For this one task, the H100 has specialized components called Tensor Cores.

A second trick is the precision of the numbers. The H100 can compute with very coarsely stored numbers, where only eight bits are occupied in memory. For AI, this level of coarseness is usually sufficient, and the calculation becomes significantly faster as a result. You can think of it like mental arithmetic with rounded values: less precise, but fast enough for the purpose.

A third point is often underestimated: memory. The H100 has 80 gigabytes of extremely fast memory located directly next to the compute core. Large models still don’t fit on a single chip, though. That’s why many H100s are linked together via a fast direct connection called NVLink and work like one single large computer.

H100 in headlines and in products

You don’t buy an H100 at an electronics store. Even so, you encounter it indirectly every day. Every chatbot response, every generated image, and every automatic translation runs on chips of this class in a data center. Providers like Amazon, Microsoft, or Google rent out computing time on them by the hour.

In business news, the H100 shows up as a unit of measurement. When a company announces it is acquiring 100,000 GPUs, that says more about its ambitions than any press release. Even nations now measure their AI capability in chip counts.

A common misconception is that the H100 is the fastest chip in the world and will remain so. Nvidia has long since introduced successors, such as the H200 and the Blackwell generation. The H100 doesn’t disappear because of this, though — it moves into cheaper price categories and continues to be used. Similar to cars, a model remains in use for years after being superseded.