Real2Sim
Real2Sim refers to methods that automatically transfer real-world objects, spaces, or scenes into a digital simulation environment. This allows robotics and AI systems to be trained in a virtual copy of reality, without having to build real hardware setups for every scenario.
To train an artificial intelligence, you need many examples. For a robot, that means many situations in which it does something and receives feedback. Generating these situations in the real world is expensive and slow — robots break things, trials fail, everything takes time. Real2Sim solves this problem by first digitally rebuilding reality. From real photos, videos, or 3D scans, a simulation environment is created in which robots or other systems can train as if they were present on site. The “Real” in the name stands for the real-world template, “Sim” for the simulation built from it.
Why simulation is cheaper than reality
In a simulation, a robot can make thousands of attempts in an hour — in the real world, that would take weeks. If something goes wrong, it costs no energy, no replacement parts, and no downtime. You can simply start over.
The decisive advantage of Real2Sim over a freely invented simulation environment is its closeness to reality. A simulation built from real measurement data has the same distances, the same surface structures, and the same lighting as the place where the robot is later supposed to work. This reduces the so-called sim-to-real gap — the gap between what a model has learned in simulation and what it can actually do in the real world. The smaller this gap, the better the trained system performs out in the field.
This is especially relevant when deploying an AI in a very specific environment: a particular factory hall, a single warehouse room, or a specific kitchen. Instead of using a generic simulation, you simply scan the real space and get a precisely matching digital copy.
How a room becomes a digital copy
The starting point is usually a scan. Depth cameras — that is, cameras that measure not just color but also the distance of every point — or conventional videos are often sufficient. Software then reconstructs a three-dimensional map of the scene from this data. Modern techniques like NeRF (Neural Radiance Fields, an AI method that computes a 3D scene from photos) or 3D Gaussian Splatting manage to preserve lighting effects and material surfaces as well, not just the rough geometry.
In the next step, this reconstruction is transferred into a physics engine — software that calculates how objects fall, roll, or tip over. Only then can a robot act within it and receive feedback. It reaches for a virtual glass that is just as heavy and smooth as the real original, learning in the process not to drop it.
A common misconception is confusing Real2Sim with simple 3D modeling. The difference is automation: with Real2Sim, the scene is created from measurement data, not by manually typing dimensions into a design program. This makes it possible to digitize many different environments in a short amount of time.
Real2Sim in robotics, research, and industry
The term is most commonly encountered in robotics research. Companies like Google DeepMind, Boston Dynamics, or the NVIDIA research lab use Real2Sim pipelines to train robots for grasping, walking, or navigating. With its Isaac Sim platform, NVIDIA has built a complete environment specifically designed for this purpose.
The principle also appears in the automotive industry: vehicles are tested in digital copies of real road sections before physically driving there. In logistics, warehouses are scanned to train autonomous transport vehicles on the exact environment before their first real deployment.
In tech news, Real2Sim is often encountered in connection with the reverse step, the so-called Sim2Real transfer: the moment when the system trained in simulation is released into the real world. Real2Sim and Sim2Real together form the cycle that drives modern robotics training today.