Background Model

Anyone who wants to find something conspicuous must first know what the inconspicuous looks like. That is exactly the task taken on by a background model. It is a mathematical description of what normally happens when nothing special happens. One example: a surveillance camera films an empty hallway. The background model records what brightness each pixel of the empty hallway usually has. When a person appears, their pixels deviate noticeably from the model — and thereby become recognizable. The term appears in very different fields: in image processing, in speech technology, in statistics, and in physics.

No outlier without a normal state

Many AI systems search for deviations. A bank looks for suspicious transfers, a security system for unusual network traffic, a sensor for the onset of a machine defect. In all these cases there are very many normal cases and very few interesting ones. A system that only knows the rare cases has far too little material to learn from.

The trick is to flip the task around. Instead of learning the exceptional case, the system learns the normal case — of which there is plenty of data. Anything that fits this model poorly is then considered suspicious. This way of thinking is called anomaly detection, that is, the detection of deviations.

The quality of the background model then determines the quality of the entire system. If it is too coarse, the system constantly raises alarms for no reason. If it is too lenient, it swallows genuine anomalies. In practice, the false alarm rate is often the bigger problem, because people eventually start ignoring warnings.

How the normal state is learned

Most often, a background model is built from statistics over many observations. With a fixed camera, the program stores a typical color value for each pixel along with a permissible range of fluctuation. If a pixel deviates more than the range of fluctuation allows, it is marked as foreground. From many such pixels, the silhouette of the person currently moving through the frame emerges.

It is important that the model continuously adapts. Light changes over the course of the day, a chair gets moved, leaves move in the wind outside. That is why good methods constantly update their background and allow several permissible states per pixel. Classic methods for this are called Gaussian mixture models; modern systems use neural networks for this purpose, that is, learning programs with many adjustable numerical values.

In speech processing, the principle looks similar, but without images. In speaker recognition, one builds what is called a Universal Background Model. It describes what an average person sounds like. The model of a single person then arises as a deviation from this average. This saves an enormous amount of recordings, because one does not have to start from scratch for each person.

From video chat to particle detector

The term is probably encountered most often in video conferencing. When a program blurs the background or replaces it with an image, it must first separate the person from the surroundings. Modern tools combine a background model with a network that detects people directly in the image for this purpose. Something similar is found in traffic cameras that count vehicles, and in checkout systems designed to detect theft.

In science, the term has a stricter meaning. Physicists at the CERN particle accelerator calculate precisely how many measurement events arise purely from already known processes. Only when significantly more events occur than this background would lead one to expect does one speak of a possible discovery. In astronomy too, this is how faint objects are separated from the noise of the night sky.

A common misconception is that a background model is simply a single photo of the empty room. That is only true of the simplest variant, and it already fails under changing light. A usable background model does not describe a fixed snapshot, but a range of permissible states. That is precisely the difference between a toy method and a system that works in continuous operation.