Demosaicing

A camera’s image sensor consists of millions of tiny light meters. Each of these measuring points can only capture brightness, not color. So that color images can still be produced, a grid of small color filters sits on top of the sensor. As a result, one measuring point sees only red, the next only green, the third only blue. Demosaicing is the computation that turns this patchwork into an image in which every pixel has all three color values. The two missing values are not measured, but estimated from the neighboring points.

Why no photo can exist without this step

Without demosaicing, a photo would have only a third of the necessary color information. The image would consist of fine red, green, and blue dots and would look like a mosaic up close. This is exactly where the name comes from: the mosaic is removed again. Every digital camera and every phone performs this step for every photo, usually without anyone noticing.

Because it involves estimation, the quality of the method directly determines image quality. Poor demosaicing produces typical errors. Colored fringes appear along sharp edges, for example a purple border around a dark branch against a bright sky. With fine patterns such as a checkered shirt, flickering color stripes appear that don’t exist in reality. This artifact is called moiré.

This is also the reason why photographers work with raw data. A RAW file contains the sensor’s measured values before demosaicing has taken place. Anyone processing it later on a computer can choose a better method than the one built into the camera. A finished JPEG, on the other hand, has already been fully processed, and this step cannot be undone.

From color grid to complete pixel

The most common filter grid is called the Bayer pattern, named after its inventor at Kodak. In it, half of all measuring points are green, and a quarter each are red and blue. This preference for green is deliberate. The human eye sees most accurately in this part of the color spectrum, so the most information is placed there.

The simplest method is pure averaging. If a point is missing its red value, the average of the neighboring red points is taken. This is fast, but it blurs edges and produces many color errors. Better algorithms therefore first check in which direction an edge runs. They then average along the edge rather than across it, because averaging across an edge would mix two different image contents.

For some years now, neural networks have taken over this task—that is, learning programs trained on millions of sample images. They have learned what real edges, hair, or fabric patterns typically look like, and fill in the colors accordingly in a plausible way. They often handle demosaicing, noise reduction, and sharpening in a single computational step. This delivers better results, but it comes with a caveat: the network invents details that look convincing but were never actually measured.

Demosaicing in phone cameras and product marketing

Every smartphone photo goes through demosaicing as one of the first steps of what is called computational photography. This refers to photography in which the software is at least as important as the lens. Whenever manufacturers advertise their image processor or an AI imaging engine, this computational step is always part of it.

The megapixel figures in advertising are also related to this. A sensor with 48 megapixels often delivers images with 12 megapixels, because four neighboring pixels of the same color are combined. This pixel binning reduces image noise and simultaneously changes the task for demosaicing. Some sensors also use color grids other than the Bayer pattern and require their own methods for this.

Beyond photography, the term also appears in industry and research. Car cameras, medical endoscopes, and satellites work with the same filtered sensors. Wherever measurement values are automatically derived from images, it matters which colors were actually measured and which were only estimated.