
Prompt Processing
Prompt Processing refers to the phase in which an AI language model fully reads in and processes the user's input before generating the first word of its response. This phase is computationally intensive and has a direct impact on how long one waits for a response.
When you ask an AI language model a question, two things happen in sequence. First, the model reads through the entire input — that is, the question, the conversation history, and any additional information — in one pass. This phase is called Prompt Processing or also Prefill. Only afterward does the model begin to respond word by word. The two phases look similar to the user, but technically they work very differently. Prompt Processing ends invisibly; what you see on the screen is already the second phase.
Why Prompt Processing determines the waiting time
The time that passes between sending a message and the first word of the response appearing is called latency. It depends almost entirely on Prompt Processing. Someone having a short conversation barely waits at all. But someone who pastes a long document in for analysis gives the model significantly more to process — and waits correspondingly longer.
This is not a flaw, but a physical limit. The model must understand every part of the input before it can respond meaningfully. Someone who uploads a 50-page contract into the chat presents the model with the same task as a lawyer who must first read that contract before answering a question about it. The difference: the model needs seconds, a lawyer needs hours — but the reading itself cannot be skipped by either.
For companies that operate AI services, Prompt Processing is a major cost driver. The longer the prompt, the more expensive each individual request. That’s why research is working intensively to speed up this phase — independent of the question of how fast the model responds afterward.
What the model actually does during this phase
During Prompt Processing, the model calculates mathematical representations for each individual word — more precisely: for each text unit, called a token. These representations capture the relationship of a token to all other tokens in the text. A model reading the sentence “The bank by the river was wet” must, for example, understand that “bank” here does not mean a financial institution. This happens through these calculations.
The results are stored in what’s called a KV cache — a kind of notepad that the model draws on during the response phase. It then doesn’t need to recompute the prompt again. Without this cache, every newly generated word would trigger the same amount of work as the entire Prompt Processing — which would be many times slower.
The distinction from the second phase, so-called Decoding or Token Generation, is important. There, the model generates the response word by word — and this fundamentally cannot be parallelized, because each new word depends on the previous one. During Prompt Processing, on the other hand, all tokens of the input can be processed simultaneously. This makes this phase faster than Decoding, but also significantly more computationally intensive per unit of time.
Where Prompt Processing becomes visible in practice
Anyone who has ever pasted a long PDF into ChatGPT, Claude, or Gemini has experienced Prompt Processing — in the form of a wait before the first word of the response appears. The longer the pasted text, the more noticeable the pause. With short questions, this phase is barely noticeable.
In products where speed is especially critical — such as real-time voice assistants or automated systems that process thousands of requests per minute — fast Prompt Processing is a decisive competitive advantage. Providers like Groq have developed specialized hardware that dramatically accelerates precisely this phase.
In technical articles and product announcements, the term frequently appears alongside metrics like “Time to First Token” — that is, the time until the first word is generated. This figure is a direct measure of how quickly Prompt Processing is completed. Anyone comparing AI services will find in it one of the most meaningful indicators of perceived response speed.