Event Stream

An event stream is an endless sequence of small messages. Each message records that something has happened: a card was charged, a video was started, a temperature sensor took a reading. Such messages are called events. They almost always carry a timestamp, i.e. the exact time of the occurrence. Unlike a table that can be read from top to bottom, a stream has no end: new items keep arriving constantly. Programs therefore don’t read the whole set at once, but process it piece by piece as it arrives.

Why companies don’t want to wait until the next morning

For a long time, corporate IT systems worked according to a simple pattern. Data was collected during the day, a large evaluation program ran at night, and the numbers were available in the morning. This procedure is called batch processing, because it works through data in large portions. That’s perfectly sufficient for monthly reports. For many of today’s applications, it’s too slow.

One example is credit card fraud. If a card is used in Hamburg and in Bangkok within four minutes, the bank must react immediately. A warning the next morning is worthless – by then the money is gone. With an event stream, the system checks every payment at the moment it is booked. Typical response times are well under a second.

This is also playing a growing role in AI. Models that predict demand, prices, or machine wear are fed with fresh events. A recommendation system that knows your behavior from yesterday is worse than one that knows your last three clicks. The value of a piece of information often decreases with every minute it ages.

From producer through the log to consumer

At the start is the producer. This is some program that reports an event: the checkout software, the app, the sensor. The producer doesn’t send the message directly to the recipient. It deposits it in a kind of intermediate storage that can be imagined as a very long notebook. New entries are always added at the bottom, existing ones are never changed.

This notebook is called a log. At the other end sit the consumers. Each of them keeps track of up to which line it has read, and works its way further forward. The clever part: multiple consumers can read the same stream independently of one another. Fraud detection, accounting, and the statistics department use the same payment events without interfering with each other.

The best-known software for this is Apache Kafka, alongside services such as Amazon Kinesis or Google Pub/Sub. They distribute the stream across many machines so that even millions of events per second can fit through. A common misconception is that an event stream is simply a fast database. The difference lies in the perspective: a database stores the current state, an event stream stores the sequence of changes. The state can be recalculated from the sequence at any time.

Where event streams run in the background of everyday life

Most event streams are never seen directly, one only notices their result. Package tracking, which continuously updates the location of a shipment, is one example. So are the live prices in a stock market app, the vehicle positions in a taxi app, or the notification from your streaming service that someone else is currently using your account.

In business news, the term often comes up in the context of trading venues and payment service providers. When an exchange advertises with latency – that is, the delay between an event and the reaction to it – stream processing is behind it. Industrial companies also use it to continuously monitor sensor data from machines and predict failures.

For you as a reader, one distinction is especially worthwhile. When a provider talks about real-time data, they usually mean an event stream. When they talk about a data warehouse or reports, it’s more likely about collected data from the past. Both approaches exist side by side in large companies, because they answer different questions.