Augmented Reality

Augmented Reality means, in English, enhanced or expanded reality. Here is what is meant: You look through a phone camera or a pair of glasses at the real environment. The device overlays additional images, arrows, or text on top of it. These additions appear as if they were really standing in the space. One example is an arrow that, in a navigation app, seems to lie directly on the street in front of you. The difference from virtual reality is important: there, everything is artificial, and the real environment disappears completely. With Augmented Reality, the real world remains the main part of the image.

What looking through the camera actually gets you

Information is more useful when it appears where you need it. A repair manual as a PDF forces the user to constantly glance back and forth between the paper and the machine. An overlaid marker points directly at the correct screw. Technicians in factories working with such systems have been shown to work faster and make fewer mistakes.

For companies, Augmented Reality is also a sales tool. Furniture stores let customers virtually place a sofa in their own living room. This reduces the number of returns, because you can see beforehand whether the piece of furniture fits. Glasses, shoes, and make-up are also tried on in this way. Every avoided return saves a retailer real money.

Many tech companies also consider Augmented Reality to be the next big device category after the smartphone. The idea: at some point, you will wear a lightweight pair of glasses instead of holding a display in your hand. Whether that will happen remains open. What is certain is that Apple, Google, Meta, and Samsung have been pouring billions into this bet for years.

How the device measures the space

For a virtual object to stay in place, the device must know where it itself is. To achieve this, two things happen simultaneously. The camera searches each frame for distinctive points, such as table edges or patterns in the carpet. Motion sensors measure, in parallel, how the device is being rotated and moved. From both of these, the software continuously calculates its own position and a rough map of the surroundings. This process is called SLAM, an abbreviation for simultaneous localization and mapping.

Only after that does the graphics come into play. The program now knows the floor surface and places the virtual sofa on it. To make it look real, it is lit appropriately and occluded wherever a real piece of furniture stands in front of it. This whole process has to be recalculated about 60 times per second. With larger delays, the object appears to float or stutter.

Artificial intelligence takes on a growing share of this work. Trained models recognize whether a surface is a floor, a wall, or a hand. Other models estimate, from a single camera image, the distance to every point in the image. Some devices additionally use a laser sensor called LiDAR, which measures distances directly. It delivers more accurate results but is more expensive and therefore not built into every phone.

From Pokémon Go to smart glasses

The best-known application was the game Pokémon Go, in which game characters appear on real streets. By now, almost every modern smartphone comes with Augmented Reality built in from the factory. Instagram and Snapchat filters that alter faces are a simple form of this. Translation apps also belong to this category: you point the camera at a sign in a foreign language, and the translated text appears in the same spot.

In the news, the term usually comes up in connection with glasses. Apple sells the Vision Pro, Meta is working on lighter models, and Google is trying again after a false start. So far, these devices are expensive, heavy, and have a short battery life. Analysts therefore regularly discuss when the mass market will arrive.

A common misconception is to equate Augmented Reality with the so-called Metaverse. The Metaverse describes networked virtual worlds, whereas Augmented Reality refers only to the overlay onto the real environment. Both terms are often mixed together in marketing. Anyone reading reports about it should check whether the real environment actually remains visible.