Streetlight Effect

The streetlight effect takes its name from an old joke. A man is searching for his keys at night under a streetlight. A passerby helps him, finds nothing, and asks whether the key was even lost here. The man replies: No, over there in the park, but the light is better here. This is exactly the pattern the term refers to: people search where searching is convenient, not where the answer lies. It’s not about stupidity, but a very understandable reaction to limited time and limited data.

Why easily measurable numbers distort our view

Anyone wanting to investigate something needs data. Some data is readily available, other data would have to be laboriously collected. So research and businesses prefer to work with what’s already there. The result is often correct, but it answers a different question than the one that actually matters.

One example from medicine: there are huge amounts of data on diseases of wealthy countries, because a lot gets documented there. There is little data on diseases that mainly affect poor regions. As a result, more studies and more medications are developed for the first group. Not because these diseases are worse, but because the light is brighter there.

The same thing happens in business with metrics. A website’s click numbers can be measured to the second, whereas customer satisfaction can only be measured approximately. So many companies steer by clicks. The well-known saying that what counts gets measured then flips around: what gets measured counts.

The mechanism behind the misguided search

The effect arises from two ingredients. First, the costs of searching are unevenly distributed: one area is cheap to investigate, another expensive. Second, there’s no check on whether the area being investigated is even the right one. Where both come together, attention almost automatically drifts toward the cheap area.

In AI development, this shows up in testing procedures known as benchmarks. These are standardized collections of tasks used to compare models, such as multiple-choice questions or coding tasks. Such tests are convenient because they yield a single percentage figure. However, this doesn’t measure whether a model is actually helpful in real everyday work. Companies optimize for the percentage anyway, because it’s the only thing that’s publicly visible.

It’s important to distinguish this from a related error. With confirmation bias, someone deliberately searches for evidence supporting their opinion. With the streetlight effect, the intent is neutral—only the location of the search is chosen wrongly. Another typical misconception is treating the effect as a pure laziness problem. Often the difficult measurement is simply unaffordable, leaving no other choice at all.

Where the effect shows up in tech news

You most often read it between the lines when new AI models are introduced. A company announces top scores across several benchmarks, yet users soon report disappointing everyday results. Critics then speak of benchmark optimization. The numbers are correct, they just measure the wrong thing.

The effect is also visible in the training material for language models. Models learn from texts that are freely available on the internet. English-language sources are massively overrepresented there. As a result, many models are noticeably weaker in smaller languages, without anyone having deliberately decided that.

For you as a reader, the effect is above all a checking tool. When a report cites an impressive number, it’s worth asking: was what was measured actually important, or just what was easy to measure? This question costs nothing and exposes surprisingly many headlines.