Schema Evolution

Every larger collection of data has a fixed blueprint. It defines which pieces of information exist and in what form they are stored. For a customer list, this might mean: name as text, age as a whole number, date of birth as a date. This blueprint is called a schema. Schema evolution is the question of how to change it later, even though millions of entries already exist and programs access it. It is therefore not a state, but a process spanning years.

Why no blueprint lasts forever

No team knows on day one all the information that will be needed later. An online shop launches without a field for the mobile phone number and needs it two years later after all. A law suddenly requires storing the timestamp of a consent. Such changes are the normal case, not the exception.

The tricky part is ongoing operation. You cannot simply switch off a database containing customer data for three days. If the schema is changed carelessly, programs crash or read values incorrectly. In the worst case, data is lost because a column was deleted that someone still needed. That is why schema evolution is considered one of the riskiest routine tasks in software development.

For AI systems, there is an additional reason. Models are trained on large data collections that have grown over years. If a field changes its meaning halfway through, the model learns from contradictory material. Such errors often only become apparent once the finished AI produces strange results.

Forward and backward compatible

The central concept is compatibility. A change is backward compatible if new programs can still read the old data. It is forward compatible if old programs can cope with the new data by simply ignoring unknown fields. Whoever maintains both can run old and new program versions simultaneously.

From this rule follow very practical rules of thumb. Adding a new field is usually harmless, as long as it has a default value. Deleting or renaming a field is dangerous. Changing the type, for example from text to number, is considered particularly risky. That is why, in practice, fields are often not renamed but a new one is created while the old one keeps running alongside for a while.

Larger overhauls therefore proceed in stages. First, the new field is created and populated in parallel. Then the programs switch over to the new field one by one. Only once it is verifiably confirmed that no one reads the old one anymore is it removed. Data formats such as Avro or Protobuf and table formats such as Apache Iceberg support this by storing the schema version alongside each data record.

Where schema evolution shows up in everyday life

The topic usually becomes visible indirectly. When an app reloads data after an update, or a bank shuts down its online banking over a weekend, a schema change is often behind it. Reports about delayed IT projects at government agencies frequently deal with this too: old data holdings cannot easily be migrated into a new schema.

In the professional world, the term comes up with tools for database migrations, such as Flyway or Liquibase. They manage changes to the schema like program code, with version numbers and the ability to roll back a step. In the data platforms of large companies, a so-called schema registry service monitors whether a planned change violates the compatibility rules.

A common misconception is that modern systems without a fixed schema solve the problem. Some databases indeed allow storing arbitrary fields. But the structure does not disappear as a result, it merely moves into the program code. There it is documented worse and errors only surface later. Schema evolution therefore does not become obsolete, it only becomes more invisible.