Prefill

When you ask a chatbot a question, the work happens in two very different phases. First, the program reads in your entire text and processes it. This reading step is called prefill. Only after that does the second phase begin, in which the answer is generated: one word after another, each calculated individually. This distinction matters because prefill can process the entire text simultaneously, while the answer phase cannot. You can picture it like a student who reads through an entire long assignment before writing the first sentence of their solution.

Why the first second determines the impression

You notice the time prefill takes directly: it’s the pause between submitting your question and the first letter of the answer. In technical terms, this waiting time is called “time to first token,” meaning the time until the first text chunk appears. For a short question, it’s in the range of fractions of a second. For an uploaded 200-page PDF, it can take several seconds.

For companies that operate such models, prefill is also a cost factor. The computational work grows with the length of the input. That’s why providers like OpenAI or Anthropic bill input text and output text separately. Input text is usually significantly cheaper, because it can be processed more efficiently.

The distinction is becoming increasingly important because requests keep getting longer. In the past, people wrote a single sentence; today, they upload entire contracts, program code, or chat histories. As a result, a large share of the work shifts from answering to reading. This is exactly why developers talk so much about this phase today.

Reading everything at once instead of word by word

A language model first breaks text down into small building blocks called tokens. A token is roughly a short word or a syllable. During prefill, all these tokens are sent through the model at once. This is possible because they are all already fixed – the model doesn’t need to guess anything.

This is exactly what makes the phase so efficient. Modern graphics cards prefer to compute many things in parallel. A prefill therefore keeps them almost fully utilized. In the answer phase, this isn’t possible: each new word depends on the previous one, so the card has to proceed one step at a time. There, the bottleneck usually isn’t computing power, but memory speed.

The result of the prefill is cached. This cache is called the KV cache – a kind of notepad containing what the model has extracted from your text. For every additional word, the model refers back to it instead of reading everything anew. Without this trick, it would have to work through the entire text so far again for every single word.

Prefill in price lists and product promises

The term is most directly visible in the price tables of AI providers. There, you’ll find separate prices for input tokens and output tokens. The input side is the prefill. Some providers additionally offer a discount for “prompt caching”: if you send the same long text multiple times, the stored notepad is reused and costs less.

The phase also appears indirectly in marketing claims. When a company advertises a “context window of one million tokens,” it is promising a very large prefill. Whether this is pleasant in everyday use depends on how fast the processing runs. A huge context window with a twenty-second wait time isn’t very useful in a chat.

A common misconception is confusing prefill with training. During training, a model learns permanently from millions of texts and changes as a result. During prefill, it learns nothing – it only reads your input for this one request and then forgets it again afterward. Your uploaded document doesn’t make the model smarter; it’s only available to it briefly.