Convolutional Layer
A convolutional layer is a component in image recognition programs that scans an image piece by piece with a small grid, thereby finding patterns such as edges or corners. Several such layers stacked one after another gradually build complex shapes from simple patterns.
Programs that recognize images consist of several processing stages. A convolutional layer is one of these stages. It places a small grid over the image, for example three by three pixels in size. This grid moves like a magnifying glass across the entire image, position by position. At each position it checks whether a particular pattern can be seen there, for example a vertical light-dark edge. The result is a new map that shows where in the image this pattern occurs.
Why image recognition wouldn’t have worked without convolution
A photo with 200 by 200 pixels has 40,000 values. If each of these values were individually connected to every unit of the next stage, billions of adjustable numbers would result. Such a program would be nearly impossible to train and would need enormous amounts of data. The convolutional layer solves this problem elegantly: it uses the same small grid for the entire image. Instead of billions of numbers, it only has to learn a few dozen.
The second advantage is even more important. A cat remains a cat, whether it sits in the top left or bottom right of the image. Because the grid searches the same way everywhere, the layer recognizes the same pattern at any position. This property is called translation invariance. Without it, a program would have to learn every shape at every possible position individually.
A common misconception: the patterns are not defined by humans. No one programs in what an edge looks like. The numbers in the grid emerge during training from thousands of example images. Humans only determine how large the grid is and how many of them there are.
From grid to feature map
The moving grid is called a filter or kernel. It contains small numbers, the weights. At each position, the layer multiplies the pixels by the corresponding weights and adds everything up into a single number. If the pattern is clearly visible there, this number becomes large. If nothing matches, it stays small or negative. The totality of these numbers forms a feature map, that is, a kind of map of the pattern being searched for.
A layer never works with just one filter. Typically 32 or 64 different filters run in parallel, each with its own weights. One reacts to horizontal edges, one to vertical ones, one to a particular color transition. In the end, the layer passes on a whole bundle of feature maps to the next stage.
Stacking is crucial. The second convolutional layer no longer sees the original image but the maps from the first one. From several edges it assembles corners and curves. The third turns these into eyes and wheels, the tenth into whole faces or vehicles. Between the layers there are usually pooling stages, which shrink the image by keeping only the largest of every four values. This way, the image section that a single unit can survey grows with each level.
Convolution in cameras, medicine, and language models
Convolutional layers are present in almost every camera app. When your phone marks a face region or automatically sorts photos by dogs and beaches, such a network is working behind the scenes. Driver assistance systems in cars also use convolution to recognize traffic signs and pedestrians. In clinics, corresponding programs search X-ray and CT scans for suspicious spots.
In business news, the term usually appears as CNN, short for Convolutional Neural Network. The breakthrough came in 2012, when such a network won a major image recognition competition by a clear margin. After that, the boom began among graphics card manufacturers, because convolutions can be computed extremely quickly on graphics chips.
It’s important to distinguish this from the Transformer, the architecture behind ChatGPT and similar text programs. The Transformer dispenses with the moving grid and instead compares all parts of an input directly with one another. For a few years now, it has been partially displacing convolution even in image processing. Nevertheless, convolutional layers remain the leading choice where little computing power is available, for example directly on a smartphone or in small cameras.