Event Stream
An event stream is a continuous chain of messages about things that have just happened – such as every click in an online shop or every executed stock order. Programs read this chain in the order in which things occurred and react immediately, instead of waiting for a later batch retrieval.
An event is a single occurrence that a computer system records: “User 4711 clicked Buy at 2:03 PM.” An event stream is the continuous chain of such messages, in the order in which they occurred. You can picture it like a receipt tape on which new slips keep arriving without interruption. Each slip describes a fact from the past and is never altered afterward. Other programs sit at the end of the tape and read along as soon as something arrives. The counter-model is a storage system that only knows the current state – that is, only how much money is in the account, but not which transactions led there.
Why companies are switching to ongoing events
For a long time, companies processed data in batches. At two in the morning, a program would run, read through the entire day, and generate reports. That is no longer enough when fraud detection needs to stop a suspicious card payment before it goes through. An event stream delivers the information in split seconds instead of the next morning.
The second advantage is decoupling. The checkout system simply writes its events into the stream and does not need to know who reads them. Accounting, warehouse management, and the recommendation system access it independently of one another. If a new application is added later, the checkout system does not need to be touched.
For AI systems, something else is important as well: training data arises right here. Every click, every search query, every correction by a user is an event. Whoever records this stream cleanly later has the material to train models or to check their quality.
What happens between sender and receiver
In the middle there is usually a system that temporarily stores and orders the events. The best-known one is called Apache Kafka. The senders are called producers, the readers consumers. The stream is divided into sections so that multiple machines can read simultaneously without getting in each other’s way.
It is important that a consumer remembers how far it has gotten. This position is called an offset. If a program crashes, it resumes exactly at that point afterward. You can also reset the offset and replay the stream again from the beginning – for instance, to correct an error in the evaluation.
A common misconception is that everything arrives immediately and is guaranteed to arrive exactly once. In practice, an event can be delivered twice if an acknowledgment gets lost. That’s why developers build their programs so that a duplicate read event causes no harm. The order is also only guaranteed within a single section, not across the entire stream.
From package tracking to the stock ticker
The principle becomes visible everywhere something is tracked live. A parcel service’s shipment tracking shows a list of events with timestamps. A stock ticker on the exchange is an event stream in which every trade generates a message. Even the notification that a friend has just posted something comes from such a stream.
In business news, the term usually appears in connection with real-time data. Banks advertise detecting fraud within milliseconds. Cloud providers sell ready-made streaming services as a product. And in privacy debates, it is often precisely these streams that are at issue, because they record in great detail who did what and when.
The event stream is to be distinguished from a classic database table. The table answers the question “What is the state now?”. The stream answers the question “What all has happened?”. Many modern systems run both in parallel and derive the current state from the stream.