Real-to-Sim

Real-to-Sim translates as “from reality to replica.” It refers to a process in which a real location is captured with cameras and measuring devices, and a computer model is built from this. In this model, one can then play through everything that would be expensive, slow, or dangerous in reality. A kitchen robot can thus reach for a cup a thousand times virtually before it touches real dishes. What matters is not only that the replica looks the same. It must also behave the same way: a table must be just as slippery, a box just as heavy.

Why robots prefer to practice on the model

Modern robot controllers learn through trial and error. They need a great many attempts for this, often millions. In the real world, this is hardly feasible. A robotic arm moves in real time, breaks down, needs power, and needs a human to reset it.

In the computer model, by contrast, trials run faster than real time and hundreds of times in parallel. What a robot would need a year to practice in a workshop, a computing facility accomplishes overnight. Accidents cost nothing either: a virtual car is allowed to crash into the virtual wall.

There is a second advantage besides. One can create situations that are rare in reality. A child suddenly running out from behind a parked car is one of the most important test cases for self-driving cars. It can only be deliberately induced in the replica.

From photo to walkable copy

At the beginning stands the capturing. One films a room from many angles, often additionally with a laser scanner that measures distances. From this data, a program computes a three-dimensional shape. Methods such as Gaussian Splatting generate photorealistic views from this, meaning images that are hardly distinguishable from a real photograph.

Then comes the harder part: the physics. The replica needs information about the weight, friction, and hardness of every object. These values appear in no photo. One estimates them, tries them out, and compares: does the real ball roll just as far as the virtual one? If the result deviates, the values are corrected. This fine-tuning is called calibration.

Nevertheless, a difference between model and reality always remains. Experts call this the reality gap. A common misconception is that one only needs to make the replica precise enough, and the gap will disappear. In practice, one proceeds differently: one deliberately randomizes the replica, making the floor sometimes more slippery, the light sometimes darker. A controller that works under all these variations will also withstand the real world. The path back is then called Sim-to-Real.

Digital twins in factories and traffic

In the news, Real-to-Sim mostly appears under the keyword digital twin. Car manufacturers rebuild entire factory halls on computers before converting a single machine. Nvidia sells software with Omniverse exactly for this purpose, Siemens and Tesla work with similar systems. This is interesting for investors because the same graphics cards are needed as for other AI applications.

In autonomous driving, the approach is standard. Companies like Waymo replay recorded road scenes on computers while altering details. This turns a single real drive into hundreds of test cases. Warehouse robots and drones, too, are almost always trained virtually first.

In everyday life, one encounters the technology unnoticed, for instance in furniture apps that project a sofa true to scale into one’s own living room. The difference from genuine Real-to-Sim: there it is only about appearance, not physics. Only if the virtual sofa also had the correct weight would it be a replica in the sense meant here.