Visual Intelligence

Visual Intelligence is a feature of newer iPhones. You point the camera at something and get information about it. For a restaurant, the phone shows opening hours and reviews, for a dog it shows the breed, for a poster it offers the option to add the event directly to the calendar. So you no longer have to type out what you’re searching for yourself — the image is the search query. Since 2025, this also works with what’s visible on the screen, not just with the camera. Apple introduced the feature in 2024 alongside the iPhone 16.

The camera as a search box

Search usually works through language. You have to be able to describe what you want to know. But that’s exactly where it often fails: What’s this plant called? What’s this part called? If you don’t know the name, you can’t search for it either. A camera completely bypasses this problem.

For Apple, the feature is also strategically important. Google makes its money from search queries, and a large share of those now happen on phones. If users increasingly take photos instead of typing, power shifts toward whoever controls the camera software. That’s why Visual Intelligence is often framed in financial news as an attack on the search business — even though Apple itself integrates Google results in the background.

Another point is usability. Older image searches first had to be opened as an app. On the iPhone 16, Visual Intelligence is tied to its own button on the side of the device. A feature you can reach in one second gets used far more often than one that requires clicking through three menus.

What happens in the background

The first step is image recognition. A program trained on millions of labeled photos assigns terms to the image: dog, storefront, text, characters. Such programs don’t learn rules, they learn patterns. They’ve seen so many photos of Golden Retrievers that they can reliably classify a new one.

After that, the system decides who answers the question. Simple cases are solved by the iPhone itself, directly on the chip. For a product search, it sends the image to Google; for open-ended questions like “What style is this?” it goes to ChatGPT, the chatbot made by the company OpenAI. According to Apple, this forwarding only happens with the user’s consent.

It’s important to distinguish this from plain object recognition, which cameras have been capable of for a long time. Recognition only tells you what’s visible. Visual Intelligence additionally tries to offer the appropriate action: call, create a calendar entry, translate text, buy something similar. A common misconception is that the phone “understands” the image the way a human does. It recognizes patterns and links them to databases — nothing more.

Competition and everyday use cases

Apple isn’t alone with this idea. Google Lens has existed since 2017 and handles very similar tasks. Samsung offers Circle to Search, where you circle an area on the screen and get results for it. Meta's camera glasses also describe what the wearer has in front of them. The technical term for this field is often “multimodal AI”: models that process not just text but also images.

In everyday life, you encounter this while traveling, shopping, and doing homework. Menus in a foreign language get translated directly. A shoe in a shop window can be found again online. A math problem in a notebook can be photographed and explained — with all the sources of error that AI answers otherwise have as well.

In the news, the term usually comes up in two contexts. First, with new iPhone generations, because Apple uses the feature as a selling point. Second, in privacy debates, because a camera that’s constantly asking questions inevitably also captures other people’s faces and license plates. How much of that the device processes itself and how much goes to servers is therefore the crucial question.