Streaming (LLM Output)
Streaming means that a chatbot outputs its answer piece by piece while it is still computing it, instead of showing everything at once at the end. That's why the text appears to be typed out on screen.
When you ask a chatbot a question, the answer usually doesn’t appear all at once. It builds up, word by word, as if typed by an invisible hand. This exact behavior is called streaming. The server doesn’t send the answer as one finished package, but as a continuous stream of small chunks. The alternative would be: you wait ten seconds staring at an empty screen and then get the entire text at once. Both take the same total time. Only the waiting feels completely different.
Why the typing text isn’t just an effect
Many people think the typing effect is a gimmick meant to fake humanness. That’s a misconception. It’s simply the technically obvious way these programs work. Language models don’t generate text as a whole, but in tiny successive steps. Streaming just passes each step along immediately instead of holding it back.
The benefit lies in perceived speed. An important metric is called time to first token. With streaming, it’s often under one second, even if the complete answer takes twenty seconds. Users rate such a system as noticeably faster, even though it computes for just as long. That’s psychology, but it determines whether a product feels sluggish or not.
There’s a second, practical advantage. You can tell early on if the answer is heading in the wrong direction and cancel it. That saves you time and saves the provider computing power. For long outputs, such as an entire page of code, this makes a real difference.
From token to chunk on screen
A language model works with tokens. A token is a piece of text, usually a short word or a syllable. The model always predicts only the next building block, appends it, and computes again. An answer of 300 words is thus produced in several hundred individual steps. Today, 20 to 100 tokens per second is typical, which roughly corresponds to reading speed.
During streaming, the connection to the server stays open while this is happening. Each finished building block is sent immediately through this open connection. Technically, this usually uses a method called Server-Sent Events, which lets the server keep sending without a new request. At the end, a signal follows indicating that the stream is complete.
A comparison makes it tangible. Without streaming, it’s like a package sent by mail: it arrives complete or not at all. With streaming, it’s like a phone call: you hear the sentence as it’s being spoken. This also explains a limitation. What has already been sent can no longer be taken back. Checks that require the finished text, for instance for prohibited content, become more difficult as a result.
Where you see streamed responses
Every widely used chat assistant uses streaming, from ChatGPT to Gemini to Claude. Coding tools that suggest code also display their suggestions continuously. Anyone building their own application will usually find a switch for this in the providers' interfaces. It’s often simply called stream and is turned off by default.
Streaming doesn’t make sense everywhere. If a program processes the answer further, for example evaluating a structured list of data, it needs the complete text. Partial data is useless there. In such cases, one deliberately waits for the complete answer.
In tech news, you mainly encounter streaming in connection with response times. Companies advertise tokens per second and very short waiting times until the first character. Streaming is not a trick that computes faster. It merely distributes the waiting time so that it’s barely noticeable.