Fallback Chain

Software that sends requests to an external service has a fundamental problem: the service can fail. It can be overloaded, respond too slowly, or return an error message. A fallback chain is the predetermined response to this scenario. It is an ordered list of alternative options that the system works through in sequence. If the first entry doesn’t work, the second is automatically tried, then the third. Ideally, the user notices nothing, because the switch happens in a fraction of a second.

Why an outage shouldn’t bring down the entire product

Most AI applications don’t run their own models. They send the request over the internet to a provider like OpenAI, Anthropic, or Google and get the response back. This interface is called an API. If it fails, the entire product grinds to a halt without any safeguard — even if the company’s own software is working perfectly.

Such outages are not rare. Major providers experience disruptions several times a year, lasting anywhere from minutes to several hours. On top of that, there are shorter bottlenecks when too many customers send requests at the same time. The provider then throttles individual customers and sends an error message instead of a response. For a company whose customer service chat depends on this, that’s costly.

A fallback chain turns a total outage into a noticeable but tolerable degradation. The responses may come from the second-best model and be somewhat less accurate. That’s almost always better than an error page. Experts call this principle graceful degradation: the system gets worse instead of simply dying.

From the first tier to the last

Between the application and the models there is usually a mediation layer, a so-called router or gateway. It knows the order of the tiers. First, the request goes to the preferred model. If no valid response arrives within a set deadline, the attempt is considered failed. This deadline is called a timeout and is often just a few seconds.

Then tier two kicks in. This can be the same model from a different provider, a weaker model from the same provider, or a small model running on the company’s own servers. At the very end, there is often a fixed fallback solution without AI: a pre-written notice or a handoff to a human. What matters is that the chain has a defined ending and doesn’t keep trying indefinitely.

Two details determine the quality. First, the system shouldn’t repeatedly run into a dead service with every single request. A circuit breaker therefore remembers that tier one has just failed and skips it for a few minutes. Second, a retry attempt must not trigger an action twice. For a plain text response, this is harmless, but for an order or a payment, it is not.

Fallback chains in real products

Anyone building an AI application quickly runs into this topic. Tools like LangChain, LiteLLM, or OpenRouter offer fallbacks as a ready-made setting: you specify a list of models, and the library handles the rest. In job postings for AI developers, the term regularly appears as an expected skill.

In the news, one usually encounters the principle indirectly. When reports describe that after an outage at a major provider, some services kept running while others didn’t, the difference often lies exactly here. The widespread strategy of not depending on a single provider is also frequently implemented technically through fallback chains.

Incidentally, the term doesn’t originate from AI. Fallbacks have existed for decades, for example with fonts: if a character is missing in the desired typeface, the browser takes the next one from a list. The fallback chain should be distinguished from load balancing. There, multiple equivalent servers run in parallel to split up requests. A fallback chain, by contrast, has a clear ranking order and is only stepped down to in the event of failure.