Regression Test

Programs are constantly being changed. A new feature is added, a bug is fixed, a piece of text is rewritten. Every such change can accidentally destroy something elsewhere that worked perfectly well before. This is exactly what is called a regression: a step backwards. A regression test is a check that is repeated after every change to detect exactly such setbacks. In doing so, one tests not just the new thing, but above all the old one.

Why old features suddenly stop working

In larger software, almost everything is connected to almost everything else. A component that formats the date is used in thirty places. Whoever changes it so that one place looks better also changes the other 29. Developers never have these connections fully in their heads — modern systems are simply too large for that.

On top of that: errors one has caused oneself tend to be searched for in the wrong place. Whoever rebuilds a payment feature tests the payment feature. That registration no longer works as a result is ultimately noticed by the customer. Such an error is significantly more expensive than one that is already noticed during programming. Rule of thumb in the industry: the later a bug is discovered, the more costly it is to fix.

With AI systems, the problem is even harder. If a language model — that is, a program that generates text — is replaced by a newer version, it can become better in many places and worse in a few. Without systematic checking, nobody notices which capabilities were lost in the process.

Test suite, automation, and the choice of cases

The foundation is a collection of test cases. A test case describes an input and the expected result: whoever enters this password is rejected. Whoever calculates 19 euros plus VAT gets 22.61 euros. These cases are written as small programs that operate the software themselves and compare the result.

The collection is run automatically, usually with every change to the program code. A server fetches the new version, runs all the tests, and reports within minutes which ones failed. Large companies run tens of thousands of such tests. Doing this by hand would be impossible, since they are repeated hundreds of times a week.

An important rule: for every bug found, a new test case is written. This way, the same bug cannot return unnoticed a second time. The test suite thus grows over time and becomes a memory of all past mishaps. For AI models, the counterpart is often called evaluation: hundreds of fixed sample questions are sent through the model and the hit rate is compared with the previous version.

From phone update to model version

Every app update on a phone has gone through regression tests before release. If a feature still stops working after an update, the test suite has a gap. Such cases regularly make the news, for example when transfers fail after a banking software update.

In tech news, the term comes up especially when AI writes program code. A model can propose changes within seconds, but it does not guarantee that everything else keeps working. The tests are the safety net here: they decide whether an automatically generated suggestion is adopted at all.

A common misconception is that a passed regression test proves error-free software. It only shows that the checked cases are correct. Anything for which nobody has written a test case remains unchecked. That is why regression tests belong to a larger bundle of procedures and replace neither the testing of new features nor the view of real users.