In the field of embodied intelligence, "Sim2Real" (from simulation to reality) has always been an inescapable challenge. Not only is the setup of simulation environments costly, but models often experience performance degradation when transferred to the real world. Recently, a new research study jointly released by Li Fei-Fei's team, NVIDIA GEAR Lab, Georgia Institute of Technology, and other institutions has provided a new approach to this problem: Real2Sim.

This system, called "SimFoundry," can generate an interactive, train-able, and test-able robotic simulation environment by simply using a video from the real world. Unlike traditional 3D scene reconstruction, SimFoundry achieves deep analysis and reconstruction of the real world, significantly lowering the barrier for building simulation environments through automation.

image.png

The core innovation of SimFoundry lies in the closed-loop construction of "digital twins" and "digital cousins." First, the system extracts video information to parse the geometric structure, physical properties, and interaction functions of objects in the scene, building a high-precision "digital twin." Based on this, the system can automatically adjust the appearance, shape, scene layout, and even operational tasks, generating a large number of "digital cousins." This means that developers only need a real-world video to obtain almost infinite training data, allowing robots to complete the entire process from strategy learning to automatic evaluation in the simulation.

Experimental data shows that SimFoundry demonstrates excellent predictive capabilities. The robot performance evaluated in the simulation is highly consistent with actual results in the real world. More importantly, robot strategies trained on these automatically generated data can achieve "zero-shot" transfer, performing excellently in complex tasks such as multi-step operations and dual-arm collaboration.

This research was co-authored by several authoritative scholars in the field of robotics, including core researchers from NVIDIA GEAR Lab and members of Li Fei-Fei's team. With the open-source release and application of this system, the development process of embodied intelligent robots is expected to undergo significant changes—no longer relying on expensive manual data collection and modeling, but instead using generative technology to enable robots to move faster from laboratories to the real world.

Paper link: https://arxiv.org/pdf/2606.28276v1