Discrete Cosine Transform

A photo consists of many small pixels, and each pixel has a brightness value. A series of such values can also be described differently: as a sum of waves that oscillate up and down at different speeds. This exact conversion is what the Discrete Cosine Transform performs. It takes in a fixed number of measured values and returns how strongly each individual wave is contained within them. The content is not lost in the process, because the original values can be reconstructed exactly from the wave strengths. “Discrete” simply means that the method works with individual measured numbers rather than a continuous curve.

Why JPEG and MP3 wouldn’t exist without it

Raw image data is huge. An unprocessed photo from a phone camera would quickly take up 50 megabytes uncompressed. As a JPEG, this often shrinks to just two or three megabytes, without the difference being clearly visible. This saving is almost entirely due to the cosine transform.

The trick lies in the fact that the conversion distributes information unevenly. In most images, brightness changes only slowly from point to point. This means that almost all the energy ends up in the slow waves. The fast waves often have values close to zero. Such values can be omitted or stored very roughly, and the image still looks good.

Without this effect, there would be no video streaming as we know it. A film in HD resolution consists of 25 or more individual frames per second. Only strong compression turns this into a data stream that a normal internet connection can carry. The H.264 and HEVC methods used by YouTube and Netflix are, at their core, based on the same idea.

From 8x8 blocks to wave strengths

JPEG first breaks an image down into small squares of eight by eight pixels each. The transform is applied to each of these squares of 64 numbers. The result is again 64 numbers, called coefficients. The first coefficient describes the average brightness of the square. The remaining 63 describe how strongly various stripe patterns are present, from coarse to very fine.

The actual data loss only happens in the next step, quantization. Here, all coefficients are divided by fixed numbers and rounded. For fine patterns, the divisor is large, because the human eye pays less attention to detail there. As a result, a great many coefficients become zero. Zeros can be stored extremely efficiently.

A common misconception is that the transform itself discards data. That’s not true: it is losslessly reversible and merely rearranges the information. Responsible for quality loss is solely the rounding that follows. Anyone who sets JPEG quality to 30 percent sees the consequences as blocky squares in the image. These so-called block artifacts reveal exactly the 8x8 grid of the transform.

Traces in everyday life and in current debates

Every JPEG on a phone, every MP3 in a music player, and every video call uses a variant of this method. For audio, instead of image squares, one works with short time segments of the signal. There, frequencies that the ear wouldn’t perceive anyway are left out, such as quiet tones right next to loud ones.

The term also comes up in the world of AI. Image generators and analysis models often work with compressed representations instead of raw pixels, because this saves computing time. Conversely, some forensic tools search for the typical traces of compression in order to detect whether an image has been edited afterward or artificially generated.

In business news, the term is usually encountered indirectly, namely through disputes over video codecs. Whoever holds the patents on a compression method earns money from billions of devices. That’s why there are royalty-free alternatives such as AV1, backed by Google, Netflix, and Amazon, among others. AV1, too, is at its core based on transforms from the cosine family.