Kernel Tuning

At its core, artificial intelligence consists of a huge number of very simple computational steps. These steps almost always run on graphics cards, i.e. on chips that can process thousands of numbers simultaneously. For a chip to know what to do, it needs a short program for exactly one computational task. Such a program is called a kernel, for example one that does “multiply these two number tables”. Kernel tuning means rewriting and adjusting such a program until it runs as fast as possible on a specific chip. The calculation itself remains unchanged; only the path to get there becomes more efficient.

Why one line of code can cost millions

Large language models spend almost their entire computation time in a handful of recurring kernels. Matrix multiplications and so-called attention together often account for over 90 percent. Speeding up these few programs by 30 percent effectively speeds up the entire system. Conversely, polishing the rest of the code brings hardly any benefit.

This has a direct impact on costs. A data center with ten thousand graphics cards burns through electricity and rental costs amounting to millions per month. If the central kernel runs a third faster, a third fewer cards are needed for the same work. That’s why AI companies employ specialists who spend months fine-tuning individual programs of this kind.

A well-known example is FlashAttention. This tuned kernel doesn’t compute attention differently, just more cleverly in terms of memory usage. Depending on the case, the result was a multi-fold speedup and significantly lower memory requirements. This is what made longer input texts affordable in the first place.

What gets adjusted when fine-tuning a kernel

The most important lever is memory, not computation. A graphics card computes extremely fast, but fetches data from its main memory comparatively slowly. Right next to the compute unit sits a tiny, very fast cache. Good tuning ensures that data is loaded there once and then used multiple times. Think of it like cooking: someone who brings all the ingredients to the counter at once is faster than someone who runs down to the cellar for each ingredient separately.

On top of that come concrete adjustable parameters. How large are the data blocks processed at once? How many computation threads run in parallel? Is the next chunk of data already being loaded while computation is still ongoing? For each of these questions there are dozens of plausible answers.

Because the combinations multiply into thousands of variants, they are often tested automatically. This procedure is called autotuning: a program measures every variant and keeps the fastest one. It’s important to note that the result only applies to a specific piece of hardware. A kernel that is optimal on an Nvidia H100 may be mediocre on an older card or an AMD chip.

Kernel tuning in the news and in everyday use

In news reports, this topic usually appears hidden. When a company announces that its model has suddenly become much cheaper at the same quality, this is often not due to a new model but to optimized code. DeepSeek also caused a stir because the team programmed very close to the hardware instead of merely using ready-made libraries.

Economically, kernel tuning is one reason for Nvidia's strong position. Over the years, the best kernels have been written for Nvidia’s CUDA programming environment. Competitors have comparable chips but less ready-optimized software. Projects like Triton are trying to close exactly this gap, making it easier to write kernels in a more hardware-independent way.

As a user, you never have to program this yourself. You still benefit from it every time a chatbot responds faster or a provider lowers its prices. A common misconception is that kernel tuning makes a model smarter. It doesn’t change the answers at all, only the speed, power consumption, and cost.