Time-to-First-Token

When you ask a chat program a question, nothing happens for a brief moment. Then the first character appears, and the rest of the answer streams in gradually. It is exactly this initial pause that Time-to-First-Token measures. It begins the moment you press send and ends as soon as the first piece of text becomes visible. A token here is the smallest unit of text that such systems work with: sometimes a whole short word, sometimes just a syllable or a punctuation mark. The time span is usually measured in milliseconds, that is, thousandths of a second.

Why the first pause determines the impression

People judge software mainly by the first moment. A response that starts streaming after 200 milliseconds feels instantly alive. The same response showing nothing at all for three seconds feels broken. Yet the second variant might even finish sooner overall. The impression, then, depends less on the total duration than on the beginning.

That’s why providers treat Time-to-First-Token as its own metric, separate from total duration. A second important figure is output speed, often given in tokens per second. It describes how fast the text continues afterward. Only both numbers together give a complete picture. A system can start blazingly fast and then continue writing agonizingly slowly.

For some applications, this metric is even business-critical. With voice assistants in cars or on the phone, users find pauses of more than about one second unpleasant. Something similar applies to a programming assistant that displays code suggestions: if the suggestion comes too late, the developer has long since kept typing on their own.

What actually happens during the wait

Before the model can output the first character, it must read in and process your entire input. This phase is called prefill. During it, the model processes the complete input text all at once and stores intermediate results that it reuses later. Only after that does the second phase begin, in which tokens are generated one after another. Time-to-First-Token essentially measures how long this reading-in process takes.

This leads to an important rule: the longer your input, the longer the initial pause. Anyone attaching a 40-page PDF waits noticeably longer than with a one-line question. The queue also plays a role. When many users make requests at the same time, the request first sits in line before any computation even begins.

Engineers reduce the waiting time with several tricks. One common approach is caching inputs: parts of a text that have already been processed once don’t need to be recomputed. Other approaches distribute the prefill phase across multiple graphics cards or provide dedicated servers for it. Physical proximity also helps, since every data packet needs time to travel across the network.

Where the metric shows up in price lists and benchmarks

Anyone integrating AI models into their own programs via an interface will find TTFT in almost every comparison test. Compute capacity providers advertise low values, often in the range of 100 to 500 milliseconds. Independent measurement services publish such figures continuously, broken down by model. Contracts for business customers sometimes even specify guaranteed upper limits.

In everyday use, you mostly notice this metric indirectly. When a chatbot stays silent for a long time on complicated requests, it’s often due to an especially long input or an overloaded data center. A common misconception is that a larger model automatically takes longer. What matters more is often the length of the input and the current load.

Product design also responds to this figure. Loading animations, blinking dots, or the gradual fading-in of text are meant to bridge the pause. This doesn’t change the measured time, but it changes the perceived speed a great deal.