Tokens per Second

When a chat program responds, it doesn’t write out its text all at once. It assembles the response from small text chunks, one after another. Such chunks are called tokens: usually a short word, a syllable, or a punctuation mark. In English, a token roughly corresponds to three to four letters. Tokens per second now tells you how many of these chunks the program manages in one second. It is therefore a speed measure, comparable to kilometers per hour for a car.

Why users and data centers watch this value

For you as a user, this value determines whether a response feels smooth. A human reads roughly five to ten tokens per second. Anything above that feels fast, because the text is generated faster than you can read it. At 3 tokens per second, the same chatbot feels sluggish, by contrast, and you find yourself waiting.

For the operator, the value is a cost question. A graphics chip in the data center costs money regardless of whether it computes fast or slow. Whoever extracts twice as many tokens per second from the same hardware effectively halves the price per response. That’s why chip companies and AI providers advertise these numbers as aggressively as car manufacturers advertise acceleration.

It’s important to distinguish this from quality. A high value says nothing about whether the answer is correct. A small, weak model is almost always faster than a large, smart one. Speed and intelligence are two separate quantities, and in the news they are often conflated.

What slows down or speeds up the pace

For each individual token, the model must be run through completely once. In doing so, billions of stored numbers, the so-called parameters, are fetched from memory and computed. The bottleneck is usually not the computation itself, but reloading these numbers. The larger the model, the more must be moved per token – and the fewer tokens per second come out.

That’s why three things help above all. First, faster memory on modern specialized chips. Second, coarser number formats: if each number is stored with only 4 bits instead of 16 bits, less data needs to be moved. Third, architectures in which only part of the model is active per token.

There are also two different perspectives to distinguish. One counts the tokens for a single user, the other counts all tokens across the entire server combined. If a server handles a hundred requests simultaneously, its total count is huge, while each individual user only sees moderate speed. So anyone reading advertised figures should check which of the two measures is meant.

Where this figure shows up in tests and price lists

In product presentations from Nvidia, AMD, or Google, it is almost always front and center. Companies like Groq or Cerebras have built their entire business around delivering extremely high values – sometimes over a thousand tokens per second for a single conversation. Comparison portals continuously measure speed, time to first token, and price for common chat services.

You notice the difference in everyday use too. A language model that runs directly on your phone might manage 10 to 30 tokens per second. A data-center-based service often delivers a multiple of that. For tools that complete your program code, every tenth counts, because otherwise the suggestions come too late.

A common misconception: some consider this figure a measure of reasoning ability. But it only describes the throughput of a machine. Numbers are only comparable if the model, chip, number format, and text length are the same. This is exactly what’s often missing in many press releases.