Shadow Evaluation
In a shadow evaluation, a new software version runs in the background with real data, without anyone seeing its results. This allows checking whether it performs better than the old version before it is actually put into use.
When a company wants to deploy a new computer program, the big question is: Does it work just as well in real operation as it did in testing? A shadow evaluation answers this question without risk. The new version runs in parallel with the old one and receives the same real requests from customers. However, its responses are only stored and evaluated, never sent to users. The name comes from the fact that the new version runs alongside the old one like a shadow. It sees everything but does nothing visible on the outside.
Why it’s better to test a system in the shadows
Test data in a lab is rarely as messy as reality. Customers make spelling mistakes, upload distorted photos, or ask questions nobody thought of. A program that achieves a 95 percent hit rate in the lab can perform significantly worse in everyday use. Shadow evaluation delivers exactly these real, messy cases.
The second advantage is safety. If the new version makes a serious mistake, no customer notices, because its response is discarded anyway. At a bank that checks transfers for fraud, a direct switchover would be dangerous. If the new system accidentally blocks thousands of real payments, the damage is immediate. In shadow operation, it remains just an entry in an evaluation table.
It is important to distinguish this from what is known as A/B testing. There, real users actually get to see different versions, usually one half the old and one half the new. This measures the effect on people, but costs real trust in case of errors. Shadow evaluation is the more cautious stage and generally comes beforehand.
What happens while running in the background
Technically, every incoming request is copied. The original goes to the established system, whose response the customer receives. The copy goes to the new version. Its result is stored in a database, together with the result of the old system. The two columns are then compared.
What is particularly interesting are the cases where the two differ. These discrepancies are often reviewed by a human, who decides who was right. For a system that automatically reads invoices, these would be, for example, all receipts where the old and new versions detect different amounts. Sometimes the old version is wrong, which a pure lab test would never reveal.
It’s not just the hit rate that is measured. Response time also counts, i.e. how long the system takes to compute per request. So do the costs, since a larger AI model consumes more computing power. A common misconception is that shadow operation costs nothing. It doubles the computing load, because every request is processed twice. That’s why often only a fraction of requests are run through it, say every tenth one.
Where shadow operation shows up in everyday life
This method is especially widespread wherever errors are costly. Banks and payment service providers use it to test new fraud detection. Search engines and online shops use it to test new rankings for search results. Manufacturers of driver assistance systems also let new software compute alongside in the car without it steering or braking. Its decisions are only compared to those of the driver.
For language models, i.e. AI systems for text, this approach has by now become standard. Before a provider switches to a new model, it lets it respond to real user queries in the background for days. This reveals whether it more frequently invents false information or responds inappropriately.
In corporate announcements, the term usually appears in connection with due diligence obligations. Companies use it to demonstrate that they tested a system under real-world conditions before deployment. For investors, this is an indication of how seriously a provider takes the reliability of its technology.