Ingestion
Ingestion refers to collecting data from many different sources and bringing it into a system that can work with it. It is the first step of any data processing – without it, an analysis or an AI simply has nothing to access.
Before a computer program can do anything with data, that data first has to arrive at it. That is exactly what ingestion means: collecting information from various sources and feeding it into a central system. The sources can be quite different – checkout systems in a supermarket, sensors in a factory, emails, PDF files, or websites. The goal is usually a large data collection in which everything sits in one place and can be searched. The word comes from the English “to ingest,” meaning to take in or swallow. Ingestion says nothing about what happens to the data afterward; it only describes the way in.
No data project without ingestion
In almost every company, data is scattered across dozens of systems. Accounting uses a different program than sales, and production yet another one again. As long as this data remains separated, no one can answer a question that spans several areas. Ingestion is the work required for such a question to even become askable.
For AI systems this is even more true. A language model that is supposed to answer questions about a company’s own documents must have ingested those documents beforehand. That is why, in many projects, ingestion is the step that consumes the most time. Experts often estimate that more than half of the effort goes into acquiring and cleaning the data – not into the actual model.
On top of that there is a simple but harsh relationship: what comes in wrong stays wrong. If records end up duplicated during intake or dates get mixed up, often no one notices later on. The resulting analysis then looks clean but is wrong. Errors at this stage are especially costly because they propagate through everything that follows.
The path from source to target system
Technically, ingestion usually consists of three sub-steps. First, a program fetches the data, for example via an interface, meaning a defined access method that another system provides. Then the data is brought into a uniform format. Finally, it is written into the target storage, often a database or a so-called data lake, a large collection storage for raw data.
There are two modes of operation. In batch processing, the process runs at fixed times, for example every night at three o’clock for the entire previous day. In streaming, by contrast, the data flows continuously, often within seconds of being generated. Streaming is more elaborate and more expensive, but it pays off wherever fast reactions matter – for instance in detecting credit card fraud.
A common misconception is to equate ingestion with data analysis. During intake, only little is changed; the actual analysis comes afterward. For AI applications, however, an intermediate step is often added: long texts are broken into sections and converted into lists of numbers so that a model can later retrieve content that is similar in meaning.
Where the term appears in news and products
Most often you encounter the word in reports about cloud providers and data platforms. Companies like Snowflake, Databricks, or Confluent make their money precisely by reliably ingesting and providing data. When “ingestion volume” is mentioned there, it refers to the amount of data flowing in per day – and thus often directly to revenue.
The term also comes up with corporate chatbots. Anyone who wants to connect a language model to their own manuals sets up an ingestion pipeline for these documents. If a manual changes, it must be ingested again, otherwise the chatbot will answer with outdated information.
And finally, ingestion appears in legal discussions. When news publishers sue AI companies, the dispute is partly about which content was ingested while collecting training data. This seemingly technical step has thus acquired a legal dimension as well.
