Robots Learn New Skills with Zero-Shot Learning: Tsinghua University, Peking University and Other Universities Jointly Propose the MotionTrans Framework

Recently, Tsinghua University, Peking University, Shanghai Jiao Tong University, and Wuhan University jointly released a human-robot collaborative training framework called MotionTrans. The innovation of this framework lies in enabling robots to learn and perform new skills by observing human actions without any demonstrations. This marks a major breakthrough in the field of robot learning.

Traditional robot training requires a large amount of real demonstration data, and collecting this data is time-consuming and costly. For example, teaching a robot how to unscrew a bottle cap requires repeated operations and recording every detail. Now, MotionTrans can capture detailed hand movement data through virtual reality (VR) devices, providing the basis for robot learning.

In the implementation of MotionTrans, researchers used portable VR devices, allowing anyone to participate in data recording at any time. The system not only records the key points of the user's hands but also synchronizes the first-person perspective video, ensuring the quality and richness of the data. After collection, the team built a dataset containing 3,213 demonstrations, covering various human-robot tasks.

The core technology lies in converting human motion data into a format that robots can understand. Researchers optimized the technology to accurately map human hand movements to the robot's joint angles, allowing the robot to perform human actions in real environments. In addition, to adapt to the robot's working characteristics, the research team adjusted speed and comfort zones to ensure the robot's safety and stability.

This innovative framework means that robots can learn more efficient skills by imitating human natural movements, laying the foundation for future human-robot collaboration.

github: https://github.com/michaelyuancb/motiontrans

Key Points:   

🦾 The research team launched the MotionTrans framework, allowing robots to learn new skills with zero samples.   

👾 Collecting human hand movement data through VR devices to build a rich training dataset.   

🔧 Human motion data is converted into robot language, achieving more efficient skill transfer.