RoboColosseum tests robot policies on real hardware at 124 sites. On the Unitree G1, none finished a task

RoboColosseum, an open initiative for testing robot AI on physical hardware, launched in early November 2026 with a preprint on arXiv (2611.02497). Participants named in the launch include Hugging Face, BitRobot, the Singapore Embodied AI Centre and researcher Jiafei Duan. It advertises more than 100 robots, including Unitree G1 humanoids, SO-101 arms, YAM bimanual systems and Franka arms, more than 140 real-world scenes, and 10,000 free evaluation hours for academic researchers.

Why it matters: robot models are mostly compared on each lab’s own hardware and protocol. RoboColosseum’s first study found that policy rankings do not carry over from one robot to another, AI Weekly reported, and that humanoid tasks remain out of reach for current open policies.

How does the evaluation work?

Teams submit a policy, which is served to robots operated locally at participating sites under standard protocols. An Open Track tests models without task-specific training; a Fine-tune Track gives each policy 100 demonstrations per task and then 25 attempts. Results are judged in blinded pairwise comparisons, and teams that apply must make their performance results public, according to Humanoids Daily.

What did the humanoid tests show?

The G1 tasks included carrying a box, folding a towel and loading clothes into a washing machine. No policy completed any of them after fine-tuning, and Duan said the open-source models were not yet suitable for zero-shot humanoid work. Partial-progress scoring was the only way to separate them. Unitree publicly backed the project and names the G1 as its humanoid platform; it did not specify any financial or hardware contribution. Other recent benchmarks include RobOmni, which adds touch to manipulation tests, and RoboHarm, which tests whether robots refuse harmful instructions.