Alibaba DAMO’s RynnWorld-Latent trains one world model on human, robot and simulation video

Researchers linked to Alibaba DAMO Academy posted RynnWorld-Latent, a world model that learns from human videos, robot demonstrations and simulation together, to arXiv on October 27, 2026. Different robots use incompatible action spaces and human videos carry no control labels; the model gets around both by conditioning on 3D-grounded “universal latent actions” from a companion model, RynnLAM.

Why it matters: if human video can be turned into robot-usable training data, the data bottleneck in robot learning eases. This is an open attempt to do that.

What data was it trained on?

Pretraining covered 23 datasets with about 553.5 million frames, mixing human interaction video, robot demonstrations and simulation rollouts, followed by post-training on target robots. The GitHub repository describes the model as a fine-tune of Nvidia’s Cosmos video model. Code is released under Apache 2.0; base weights are downloaded separately.

How does it make training data?

Two ways. “Digital teleoperation” generates action-video demonstrations from target-robot commands. Latent-action synthesis transfers motion from a source video onto an edited first frame and pairs the result with retargeted pseudo robot actions. The authors report better video-prediction metrics on public benchmarks and two real robots, improved policy success on LIBERO and LIBERO-Plus after synthetic mid-training, and better real-world success than real demonstrations alone in the settings tested. The abstract gives no percentages.

Related: the world-models bet and DeepMind’s Genie 4.