Change Data Feed

Large companies store their data in tables, that is, in large lists with rows and columns. Such a table can have millions of rows, for example all the orders of an online shop. It is constantly changing: rows are added, others are corrected or deleted. A Change Data Feed is an additional data stream that logs exactly these changes. Instead of saying “this is what the table looks like now,” it says “this is what has changed since yesterday.” Other programs can read this stream and update their own copies of the data without reloading everything.

Why nobody wants to reload the entire table

Data in companies rarely lives in just one place. The same order data ends up in a management report, in a search function, and in a model meant to detect fraud. All these copies must stay up to date. Without a change log, there’s only one crude way: recopy the entire table every night.

That’s expensive. Cloud providers bill based on the amount of data processed and on compute time. If only 200,000 out of 500 million rows have changed, you’re processing 99.96 percent of the data completely for nothing. A Change Data Feed reduces the work to the part that actually changed. In practice, this often cuts runtimes from hours down to minutes.

The second benefit is freshness. Anyone who only fully copies at night has outdated numbers during the day. With a change stream, a system can catch up every few minutes. For a dashboard meant to show revenue in near real time, that’s the decisive difference.

What’s in a change row

The feed itself consists of rows again. Each row contains the actual data plus some additional information. The most important is the type of change: inserted, deleted, or a value was overwritten. In the case of an overwrite, many systems deliver two rows, the state before and the state after. This allows a downstream program to trace exactly what happened.

On top of that comes a version number and a timestamp. Table formats such as Delta Lake, Apache Iceberg, or Apache Hudi number every write operation sequentially. A reading system remembers up to which version it has gotten. On the next run, it simply asks: What has happened since version 4711? The principle resembles a bookmark in a book.

Technically, this is possible because these formats never actually overwrite data. They write new files and keep a log of which files belong to which version. At its core, the Change Data Feed reads out this log. Related but not identical is Change Data Capture: here, one taps into the change log of a classic database, such as PostgreSQL. The purpose is the same, the technical location is different.

Where the term appears in products and news

It is most often read in the context of Databricks and the Delta Lake format. There, Change Data Feed is a setting that you switch on per table. After that, the change stream is available via a normal query. Competing formats such as Iceberg and Hudi offer comparable functionality under their own names.

In everyday life you don’t notice this directly, but you do feel its effects. When a banking app shows a transaction within a few minutes, such a change pipeline is often behind it. Recommendations in shops or streaming services are also updated via change streams instead of being fully recalculated once a day.

The topic is gaining importance for AI applications. A search system that searches company documents must reprocess changed documents. The change stream tells it which ones those are. A common misconception, by the way, is that a Change Data Feed is a complete archive. It is not: old entries are deleted after a configured period, often after 30 days.