Lakehouse

Large companies collect enormous amounts of data: sales figures, website clicks, sensor readings from machines, photos, text files. This data has to be stored somewhere, and it must be possible to analyze it later. Until now, there were two separate approaches for this. Either everything was dumped unsorted into a huge, cheap storage system, or everything was tidily organized into tables beforehand, which was expensive and laborious. A lakehouse is an attempt to have both at the same time: cheap mass storage that nonetheless holds ordered tables you can search directly. The name is a made-up word combining the English terms for data lake and data warehouse.

Why companies no longer want to store their data twice

The unsorted mass storage is called a data lake in technical jargon. It is very cheap and accepts everything, regardless of format. The problem: nobody guarantees that the data is clean. The same customer number might appear three times, columns may be missing, formats may not match up. Analyses on top of it are tedious and slow.

The ordered counterpart is called a data warehouse. There, data sits in fixed tables with clear rules. Analyses run quickly and reliably. In return, storage space is expensive, and every type of data has to be trimmed to fit beforehand. Images, videos, or raw sensor data often don’t fit in there at all.

Many companies therefore operated both in parallel and constantly pushed data from the lake into the warehouse. This costs twice as much, and over time the two data sets drift apart. A lakehouse is meant to make this duplicated effort unnecessary. This is especially important for AI, since models need both clean tables and raw text and images.

The metadata layer above the files

Technically, a lakehouse sits on completely ordinary, inexpensive cloud storage. There, many files simply reside in a space-saving format. That alone would just be a data lake. The crucial addition is a management layer on top, known as a table format. Well-known examples are called Delta Lake, Apache Iceberg, and Apache Hudi.

This layer keeps track of which files belong to which table and what columns exist. You can think of it like a library catalog. The books sit on cheap shelves in the hall, but the catalog knows exactly what is where and in which edition. Without the catalog, you would just have a pile of paper.

This gives the cheap storage capabilities otherwise only known from databases. Changes are either applied completely or not at all, so half-finished states are never visible. You can jump back to older versions of a table, which is called time travel. And multiple programs are allowed to read and write simultaneously without destroying each other’s work.

Databricks, Snowflake, and the data teams

The term was popularized above all by the company Databricks, which published a widely cited paper on the topic in 2020. Its competitor Snowflake came from the data warehouse side and is now moving in the same direction. Amazon, Microsoft, and Google also offer lakehouse services. In business news, the term therefore often comes up in connection with acquisitions and billion-dollar valuations of these providers.

In everyday life, one doesn’t encounter a lakehouse directly, only through its results. When a streaming service computes recommendations or a bank detects suspicious payments, the underlying data often resides in such an architecture. Training data for AI models is also prepared and versioned there.

A common misconception is that a lakehouse is a single product you can buy. It is rather a blueprint made up of several building blocks: storage, table format, query tool. And it doesn’t automatically solve all problems. Anyone who loads bad data in will only get bad analyses out of a lakehouse as well.