Durable Workflows

A program normally runs only in a computer’s working memory. If that computer fails, everything that happened up to that point is lost. For a short operation, that’s not a big problem: you simply restart. For an operation that takes hours or days, restarting from scratch would be costly and frustrating. Durable Workflows solve exactly this problem. After every intermediate step, they permanently record what has already been done, and can therefore continue at the same point after an interruption.

Why crashes in the middle of a process are so costly

Many business processes consist of several steps that must happen one after another. An online order charges money from the card, reserves the goods in the warehouse, books shipping, and sends a confirmation. Between these steps, sometimes minutes pass, sometimes days. If the server crashes after the payment has been charged, a dangerous intermediate state arises: the customer has paid, but no one ships anything.

Without special precautions, such cases have to be laboriously cleaned up by hand. Developers then build their own database tables, retry logic, and cleanup scripts. This code has nothing to do with the actual business logic and yet is often the largest part of the program. Durable Workflows take this work off your hands, because storing and resuming is handled by the platform.

A second reason is traceability. Because every step is recorded, one can later look back exactly at what happened and when. This helps with debugging and with audits, for instance when a bank must document why a payment proceeded the way it did and not otherwise.

The log from which the process is restored

At the core is a log, technically called an event log. Every time the process does something important, the result is written to a database. If the process restarts after a crash, the platform reads this log from the beginning. Steps whose result has already been recorded are not actually executed again, but merely replayed. Only at the first unknown point does real work resume.

You can picture it like a cookbook with checkmarks. After each completed work step, a checkmark is placed in the margin. If someone new comes into the kitchen, they read the checkmarks and immediately know where to continue. The cake doesn’t get put in the oven twice.

For this to work, the process must be deterministic: given the same inputs, the same result must always come out. Random numbers or the current time therefore cannot simply be used directly, but only through platform functions whose results are likewise logged. Calls to the outside world, for example to a payment interface, run as so-called activities. They are automatically retried on failure, with growing wait times in between.

From order systems to AI agents

Well-known tools of this kind include Temporal, Azure Durable Functions, AWS Step Functions, and Restate. Major providers such as Netflix, Stripe, or Uber use such systems for billing, deliveries, and order processes. In job postings and technical articles, the term Durable Execution also appears, referring to the same principle.

The topic is receiving particular attention because of AI agents. These are programs that query a language model multiple times in succession while using tools, such as a search engine or a database. Such chains take a long time, cost money with every model call, and frequently fail. A Durable Workflow ensures that after an error, the entire expensive chain doesn’t have to be recomputed from scratch.

A common misconception is equating Durable Workflows with a queue. A queue transports individual tasks but does not know the overall process. A Durable Workflow knows the sequence, the state, and the error handling of the whole process. Queues nevertheless remain important, since they are often used internally within such platforms.