Graph-Based Pipeline

Many tasks on a computer consist of several work steps that build on one another. You can arrange them as a chain: step one, then two, then three. A graph-based pipeline arranges them as a network instead. Each work step is a point in this network, and an arrow from one point to the next means: this step needs the result of the previous one. Steps between which no arrow runs are not dependent on one another and may therefore run at the same time. Experts call such a network of points and arrows a graph — hence the name.

What the network has over the rigid chain

A chain is simple, but wasteful. Imagine a recipe that says: peel potatoes, boil potatoes, wash salad, arrange salad. Read as a chain, you stand next to the pot for twenty minutes before you touch the salad. In reality, the salad doesn’t depend on the potatoes at all. A graph makes exactly this visible: two separate strands that only come together at the end when serving.

With software, it’s about the same kind of saving, just on a larger scale. When training an AI model, huge amounts of data must be cleaned up beforehand: reading in texts, removing duplicates, detecting languages, filtering out unwanted content. Some of this can run in parallel on hundreds of machines. Whoever notes these dependencies cleanly saves hours of computing time and thus real money.

The second major advantage is troubleshooting. If a step fails, the system knows exactly which later steps are affected by it. It doesn’t have to start everything over from the beginning, but only needs to repeat the broken part. For processes that take several days, that’s the difference between a hiccup and a catastrophe.

How the order arises from the arrows

The programmer doesn’t write down when which step is supposed to run. They only describe which step needs which inputs. From this information, a control program — usually called an orchestrator or scheduler — assembles the graph. It then searches for all points whose preconditions are met and starts them. Once a step is finished, it checks again what has now become possible.

One rule is mandatory here: the arrows must not form a cycle. If step A waits for B and B waits for A, no one is waiting meaningfully anymore and the process comes to a standstill. That’s why it’s called a directed acyclic graph, or DAG for short. Directed means the arrows have a direction. Acyclic means there are no loops.

A common misconception is that a graph-based pipeline automatically makes everything faster. That’s only true if there are actually independent branches. If the process is a true chain in which each step really waits for the previous one, the graph brings no time savings. It remains useful nonetheless, because it documents the process and makes restarts easier.

Where such processes work in the background

Graph-based pipelines are most visible in tools bearing names like Airflow, Dagster, or Prefect. Such programs control nightly data evaluations in companies, for example the aggregation of sales figures from different countries. Even when developers submit new program code, a graph runs in the background: compile, test, package, deliver.

In the AI industry, the term appears especially in data preparation and in connecting multiple models together. A voice assistant, for instance, can simultaneously start a search query, check the calendar, and load the user history, only combining the results afterward. In product announcements, you then read about workflows or orchestration — usually meaning exactly this principle.

One should not confuse the term with a knowledge graph, even though both contain the word graph. A knowledge graph stores facts and their relationships, for example that a company is headquartered in a particular city. A graph-based pipeline does not store facts, but work steps and their order.