Event-Based Sensors

An ordinary camera works like a flip book. It captures many complete images per second, typically 30 or 60. Each of these images contains every pixel, including those where nothing has changed since the last frame. An event-based sensor does the opposite: each of its pixels observes brightness independently. Only when it changes significantly does that pixel report with a brief message. The result is not a film, but a constant stream of messages of the type: at this location, at this moment, it got brighter.

What cameras couldn’t do until now

The biggest advantage is speed. A pixel of an event-based sensor reacts within a few millionths of a second. A regular camera, by contrast, has to wait until the next frame is due. A ball flying at 100 kilometers per hour leaves a blurred streak on a regular photo. An event-based sensor, on the other hand, traces its motion cleanly, point by point.

The second advantage is data volume. In many scenes, only a small part of the image is moving. A surveillance camera in an empty hallway still delivers full images hour after hour. An event-based sensor stays almost completely silent during that time. Less data means less storage, less computing effort, and significantly less power. Some of these sensors run on a fraction of the power a normal camera needs.

There’s also the wide dynamic range. Because each pixel regulates its own exposure, the sensor copes well with strong contrasts. Driving out of a dark tunnel into bright sunlight often overwhelms regular cameras for a moment. And it’s exactly in that moment that a driving assistant might miss an obstacle.

A design modeled on the retina

The model is the human eye. Our retina, too, doesn’t send complete images to the brain, but mainly reports changes. That’s why such sensors are sometimes called Silicon Retina, i.e., a retina made of silicon. Each pixel has its own small circuit that compares the current brightness to the last reported value.

If the difference exceeds a set threshold, the pixel triggers what’s called an event. This event consists of only four pieces of information: the column position, the row position, the timestamp, and the direction of the change. Brighter or darker — no more information is included. The pixels work independently of one another and without a shared clock.

This is exactly what makes processing demanding. Standard image recognition programs expect rectangular images as input and can’t do anything with an irregular stream of points. One must either compute the events into artificial images over short time windows or use specially adapted models. Another common misconception is that an event-based sensor is simply a faster camera. In a completely static scene, it delivers almost no data at all, because nothing is changing.

From the factory floor to the smartphone

The technology is most advanced in industry. In factories, such sensors inspect components that pass a camera very quickly, or monitor machines for subtle vibrations. They are also popular in robotics and drone research, since a flying device needs to react very quickly to obstacles while having little power to spare.

In the news, event-based sensors often come up in connection with neuromorphic technology. This refers to hardware modeled on the structure of the brain. Manufacturers such as Sony, Samsung, and the startup Prophesee are working on such chips. In smartphones, they are being discussed as an add-on that compensates for camera shake or speeds up autofocus.

Another field is data privacy. Because the sensor doesn’t deliver sharp faces but only outlines of movement, it can be used to monitor spaces without making people clearly identifiable. Whether that is legally sufficient, however, is disputed. Overall, for now the technology remains a complement to the regular camera, not a replacement.