Streaming Renderer

When a chat program answers, the response doesn’t appear all at once. It is computed word-part by word-part and sent to the screen in small chunks. A streaming renderer is the part of the program that makes these chunks visible immediately. Rendering here means: turning raw text into a fully formatted display, complete with headings, lists, and highlighted code sections. The streaming renderer does this continuously anew, even though the text is not yet complete. That’s why it looks as if someone is typing live.

Why the feel of typing determines usage

A complete answer can take ten or twenty seconds. Staring at a blank screen for twenty seconds feels very long. If, however, the first word appears after half a second, the same answer feels fast. Experts call the time until the first visible character the “time to first token.” It is often more important for the user experience than the total duration.

There is also a practical advantage: you can stop early. If, after two lines, it’s clear the answer is off-topic, you click stop. That saves the provider computing time and thus money. Without streaming, the request would be computed to completion before you could even see what comes out.

But there is also a downside. The user reads along while the model is still writing. A false statement can already be on screen before any checking mechanism could stop it. Anyone wanting to check responses for problematic content beforehand must partially slow down streaming.

From chunk to formatted display

The small chunks are called tokens. A token is a piece of text, usually a short word or a syllable. The server sends them individually over an open connection that remains in place for the entire response. A common technique for this is so-called Server-Sent Events, a simple standard for such data streams. The display in the browser appends them to the text so far.

Formatting is where it gets tricky. Most chat programs use Markdown, a simple markup language: two asterisks around a word mean bold text, three backticks start a code block. During streaming, though, often only the first asterisk arrives, with the second following later. The renderer must therefore decide what to do with half-finished instructions.

Good implementations therefore work with a kind of buffer. Incomplete character sequences are provisionally shown as plain text or briefly held back. As soon as the closing character arrives, the section is rebuilt. If the text visibly jumps back and forth in the process, this is called layout flicker. It is considered a sign of a poorly built streaming renderer.

Where text appears word by word

The effect is best known from ChatGPT, Claude, Gemini, and similar services. Coding assistants like GitHub Copilot Chat and search functions with AI summaries also work this way. Anyone building their own application will find a switch called “stream” in almost every AI interface. If it’s set to true, the response arrives as a stream; otherwise, as a single package.

In news about AI products, the term usually comes up in connection with perceived speed. Providers advertise low latency, meaning a short delay until the first character. A common misconception is that streaming makes the model faster. It doesn’t: the total computing time stays the same, it’s just packaged differently.

Streaming doesn’t make sense everywhere. If a program processes an answer further in the background, for example as structured data for a database, displaying it step by step accomplishes nothing. In that case, it’s better to wait for the complete result. Streaming pays off especially where a human is watching directly.