Durable Streams

In many computer systems, data doesn’t arise all at once, but bit by bit. Every card payment, every click, every temperature reading is a single small event. Such a continuous sequence of events is called a data stream. With simple data streams, an event is gone as soon as the recipient has read it. Durable Streams do things differently: they additionally write every event to a hard disk and keep it there for hours, days, or years. As a result, the stream can later be replayed from the beginning again, or from any arbitrary point.

Why a lost event becomes costly

The most important reason for durability is resilience to failure. Servers crash, networks go down, software gets restarted. With a transient data stream, the events from those minutes are irretrievably lost. With a Durable Stream, they remain on disk. The program remembers how far it got, and afterward simply keeps reading from there.

A second reason is the decoupling of sender and receiver. An online shop can write orders into the stream even while the invoicing software is currently offline for maintenance. The orders wait patiently until the receiver is back. Without durable storage, the sender would either have to block or discard data.

On top of that, there’s a practical advantage in developers' everyday work: replaying. If an evaluation contained a bug, you fix the program code and run the old events through it once more. The result is completely recalculated. With transient streams, this second chance simply doesn’t exist.

The logbook with a running number

Technically, a Durable Stream is usually what’s called a log, i.e., a protocol/record. New events are only appended at the end, never changed in the middle. Every event gets a running number, the offset. You can picture this like a ledger in which every line is numbered and no one is allowed to erase anything.

The receiver only needs to remember a single number: the offset up to which it has read. After a restart, it asks the stream for everything from that number onward. Multiple receivers can read the same stream independently, each with its own position. This is exactly why fraud detection doesn’t interfere with accounting, even though both process the same payment events.

So that a single server doesn’t become a bottleneck, the stream is split into sections distributed across different machines. In addition, each section is copied multiple times, typically threefold. If a hard disk fails, a copy takes over. Durability here therefore doesn’t just mean storing, but storing multiple times in different places.

From Kafka to the AI agent

The best-known system of this kind is called Apache Kafka; similar services are offered by Amazon, Google, and Microsoft in their data centers. Banks send transactions through it, streaming services their click data, logistics companies the positions of their trucks. Anyone who reads about event streaming in job postings or quarterly reports almost always means this technology. It rarely becomes visible to users—it works in the background.

In the AI world, the term has recently taken on a second meaning. Language models output their answer word by word, and programs that work autonomously over many minutes produce long sequences of events in the process. If the connection drops, half the answer would be lost without storage. Providers therefore set up durable streams that a client can reattach to after a disconnection.

A common misconception is confusing Durable Streams with a normal database. A database stores the current state, for example today’s account balance. A Durable Stream stores the events that led to that state. The two complement each other: the state can always be recalculated from the stream, but not the other way around.