Immersive Navigation

Classic navigation shows a map from above. You see an arrow marking your own position and have to mentally transfer this map onto the real surroundings. Immersive navigation reverses this. The cues appear directly in the image you’re already looking at anyway: on the phone’s camera view, on the car’s windshield, or in a data headset. A green arrow then appears to lie on the street and turns exactly where you’re supposed to turn. The device takes over the work of translating between map and reality.

Why arrows on the street produce fewer mistakes

The biggest benefit is the reduced mental load. Anyone reading a map has to constantly translate: north is up, but I’m currently facing south. This mental rotation costs attention that’s in short supply while driving or in a crowded train station hall. If the cue lies directly on the actual path, this translation step disappears entirely.

This becomes especially clear at confusing intersections. In a city there are spots where five streets meet and the instruction “turn right” is ambiguous. An arrow that lies exactly on one of the five streets is unambiguous. Studies from map providers cite significantly fewer wrong turns as the main argument here.

Economically, the term is interesting because it acts as a connecting link. Car manufacturers, headset makers, and map services all need the same foundation for this: extremely precise three-dimensional data of the surroundings. Whoever owns this data holds a strong position. That’s why reports on immersive navigation often surface in connection with acquisitions of mapping companies.

From positioning to the overlaid arrow

The first step is the question: where exactly am I? Satellite positioning via GPS isn’t sufficient for this. In cities it’s often off by ten meters or more, because signals bounce off high-rise buildings. But a ten-meter error means the arrow ends up on the wrong sidewalk.

That’s why a second method is added. The camera captures the surroundings, and the software compares the image with a stored collection of images of the same street. If it recognizes the same building facade, it knows the position to within a few centimeters. This recognition of places from images is called visual localization. In addition, motion sensors in the device measure which direction the gaze is turning.

Only after that does the drawing happen. The system now knows location and viewing direction and knows from a three-dimensional model where the street runs. It calculates which point on the screen the arrow belongs to so that it appears to lie on the ground in correct perspective. Occlusion is also important: if a bus is standing in front of the curve, the arrow should disappear behind the bus. Without this effect, the overlay looks like a sticker on the glass.

Phone cameras, head-up displays, and smart glasses

The best-known variant is the pedestrian version in map apps. You hold up the phone, the camera activates, and directional arrows and distance readouts float over the image of the street. Google calls this feature Live View, and Apple offers something comparable in its Maps app. It’s usually used briefly, for instance right after leaving a subway station, because holding up the device gets tiring over time.

In cars, it works via a head-up display, meaning a projection onto the windshield. Models from Mercedes, BMW, and several Chinese manufacturers overlay turn arrows so that they appear to lie on the road a few meters ahead of the vehicle. The gaze doesn’t have to leave the road.

The third environment is glasses and headsets. Here, immersive navigation is considered one of the few applications that could justify the effort. A common misconception, however, is that any map shown in a headset is already immersive. What matters is not the device, but whether the cues are firmly anchored to the real surroundings. A floating mini-map alongside is precisely not that.