Machine Vision

To a computer, a photo is initially just a huge table of numbers. Each pixel is described by values for red, green, and blue. A phone image can easily have twelve million such points. Machine vision is the field that turns these numbers into a statement about the image content. That statement might be: “This is a bicycle,” “This screw has a crack,” or “This face belongs to the device’s owner.” The English technical term for this is computer vision, and it is also frequently used in German texts.

Why images are so hard for machines

Humans recognize a dog in a fraction of a second, whether it’s sitting, running, or half hidden behind a car. This ability seems self-evident, but it is extremely demanding. The same object looks completely different under different lighting, from a different angle, or at a different distance – expressed in numbers, the resulting image bears almost no resemblance to the first one. For decades, programs failed at exactly this task.

Since around 2012, this has changed dramatically. That year, a learning system won a well-known image recognition competition by a wide margin. For certain clearly defined tasks, computers today are as reliable as trained humans. In medicine, such systems mark suspicious areas on X-ray images, for example, so that doctors can look more closely in the right places.

Economically, this field is important because cameras are cheap. A sensor that can see costs only a few euros and can be mounted almost anywhere. This makes it possible to automate inspection tasks that used to require people. This is precisely where the great opportunities lie – and equally great debates about surveillance.

From pixel to edge to object

The systems common today learn from examples rather than fixed rules. They are shown a very large number of images that are already labeled with what can be seen in them. The system compares its guess with the correct answer and then adjusts its internal computational values slightly. After millions of such rounds, the guesses usually fit.

Technically, this happens in several layers stacked on top of one another. The first layer slides small search patterns over the image and responds to simple things: jumps in brightness, edges, color changes. The next layer combines these edges into corners and curves. Further up, these become wheels, eyes, or window frames, and at the very top, finally, “car” or “cat.” This structure is called a convolutional neural network, because sliding the search patterns across the image is mathematically a convolution.

Several types of tasks can be distinguished. In classification, the system provides only a single word for the whole image. In object detection, it additionally draws boxes around each object found. In segmentation, it decides for every single pixel what it belongs to – something a car needs in order to distinguish the road from the curb.

Cameras that already think along today

The closest example is your own smartphone. Face unlocking, searching for “beach” in the photo gallery, and blurring the background in a video chat are all machine vision. Translating a photo of a menu also belongs here: first the text in the image is found, then read, then translated.

In industry, cameras on assembly lines check workpieces for scratches, often several parts per second. Checkout systems recognize types of fruit, agricultural machines distinguish crops from weeds and spray them selectively. Driver assistance systems read traffic signs and brake in front of obstacles.

A common misconception is that such systems understand images the way a human does. They recognize statistical patterns without any knowledge about the world. That is why they sometimes mistake a printed T-shirt for a real object, and why error rates differ for groups of people who were rarely represented in the training data. In the news, the term therefore often appears in two senses: as technical progress, and as a question of privacy and fairness.