Data Lineage
Data lineage refers to the seamless tracking of where a dataset originates, which stages it has passed through, and how it was altered along the way. Companies and AI developers use it to detect errors in data and to fulfill legal proof obligations.
Imagine you receive a spreadsheet with sales figures — but nobody knows anymore where the data came from or whether someone edited it along the way. Data lineage solves exactly this problem. It is the complete record of a dataset’s life journey: Where was it created? Which systems did it pass through? Who changed it and when? The result is a kind of family tree for data, documenting every step — from the first entry to the final analysis.
Why faulty data is so dangerous
In modern companies, data passes through many hands and systems. A customer database feeds reports, the reports flow into AI models, and the AI models influence decisions. If an error creeps in at any point, it propagates through the entire chain.
Without data lineage, it’s like searching for a printing error without knowing which print shop produced the book. With data lineage, you can see at a glance: in step three, the revenue column was overwritten by a faulty conversion. This saves hours of troubleshooting and prevents incorrect figures from feeding into important decisions.
For AI models, this is especially critical. A model is only as good as the data it was trained on. If training data is of unclear origin, it’s impossible to verify afterward whether it was biased or flawed. Data lineage makes this kind of verification possible.
How a dataset’s family tree comes into being
The basic idea is simple: every time a dataset is created, copied, filtered, or merged, the system leaves a record. This record contains at least three things — what happened, when it happened, and which system or person was responsible. Put together, these records form a directed chain that can be read backwards like a bank statement.
In practice, there are two approaches. With manual capture, teams maintain this information themselves in documents. This is labor-intensive and error-prone. With automatic capture, specialized tools — so-called lineage tools — monitor data movements directly within the systems and write the log without human intervention. Major platforms like Apache Atlas or cloud services from Microsoft and Google offer such functions built in.
An important distinction exists between the coarse and the fine level. At the coarse level, you can see that file A flowed into system B. At the fine level, you can see that column three from file A was merged with column seven from file C, and that the lower value was chosen in the process. The finer the documentation, the more useful it is — but also the more effort it requires.
Data lineage in regulation and AI development
Banks and insurers in Europe are legally required to demonstrate how their risk metrics are derived. European banking supervision codifies this under the abbreviation BCBS 239. Without data lineage, companies simply cannot provide this proof — and risk severe penalties.
The EU AI Act, which has been gradually taking effect since 2024, goes a step further. For so-called high-risk AI systems — for example in credit lending or personnel selection — companies must document which data their model was trained on. Data lineage is thus no longer an optional quality feature, but a legal requirement.
The term also comes up regularly in reporting when companies must explain why an AI system made a faulty or discriminatory decision. It usually turns out that the origin of the training data was not properly documented. Data lineage is meant to prevent exactly that.
