Tokens per Second
Tokens per second measures how fast an AI language model outputs text. The number indicates how many small text chunks the model produces in one second – the higher the value, the more fluid the response appears.
When an AI chat program responds, the text usually appears on the screen piece by piece. These pieces are called tokens: short text chunks, often a word, a syllable, or a punctuation mark. Tokens per second indicates how many of these chunks the program outputs in one second. This figure is thus a measure of speed, similar to kilometers per hour for a car. At 50 tokens per second, roughly 35 to 40 words are produced in the same amount of time. That is significantly faster than a human can read.
Why this number determines the user experience
For users, speed is a large part of quality. A model that gives smart answers but takes a minute to do so feels tedious. As a rough rule of thumb: from about 10 tokens per second onward, one can keep up while reading, and from 30 onward the output appears fluid. Anything below that feels sluggish, even if the content is correct.
For providers, the number is primarily a matter of cost. A computing chip that achieves twice as many tokens per second can serve twice as many requests. This halves the cost per response. That is why tokens-per-second figures almost always appear in product announcements from chip manufacturers and AI companies.
An important distinction: tokens per second says nothing about the quality of the answer. A small, weak model often achieves very high values because it has less to compute. Speed and capability are two separate properties. Anyone comparing models must consider both numbers side by side.
What limits the speed
A language model does not generate text all at once. It computes one token, appends it to the text so far, and starts over. For every single token, the chip must pass through the entire model once. This loop cannot be skipped, because each token depends on the previous one.
The bottleneck here is usually not the computation itself, but reading from memory. The numerical values of a large model occupy many gigabytes. They must be fetched from graphics memory for every single token. How fast this happens depends mainly on the chip’s memory bandwidth – that is, how much data can flow through per second.
That is why two tricks are especially helpful. First: make the model smaller, for instance by storing its numbers more coarsely. Second: activate only part of the model per request, as Mixture-of-Experts models do. Both reduce the amount of data per token and thus directly increase tokens per second.
Where the figure appears in tests and product news
In comparison tests of AI services, the number usually appears alongside two other values. One is the time to first token, i.e., how long it takes until anything appears at all. The other is the price per million tokens. Together, these three values give a picture of speed, response time, and cost.
Anyone running a model on their own computer sees this figure as well. Programs for local AI display, after each response, how many tokens per second were achieved. On a normal laptop, this is often only a few, while on a powerful graphics card it is considerably more. This quickly shows which model size one’s own hardware can handle.
A common misconception: companies sometimes cite huge figures like several thousand tokens per second. These numbers almost always apply to many users simultaneously, not to a single response. One must therefore distinguish between a server’s total throughput and the speed a single user experiences.