A new model, WholeBodyWAM, has been developed to control humanoid robots capable of moving and manipulating objects simultaneously. The system extends the capabilities of existing World Action Models (WAMs), which previously focused mainly on manipulation tasks using only the arms or on tabletop operations. WholeBodyWAM combines predictions of visual dynamics, manipulation actions and intentions for controlling the robot’s whole body, making it applicable to a broader range of tasks.
How WholeBodyWAM Works
The core idea behind WholeBodyWAM is to model visual dynamics and robot actions jointly. Unlike previous work, which focused solely on manipulation, WholeBodyWAM accounts for the full range of a humanoid robot’s movements. The system retains pre-trained models of world interactions and actions while adapting them to the specific requirements of whole-body control. This is achieved by integrating the semantics of heterogeneous whole-body controllers (WBC) and coordinating their operation.
This approach allows the model to make effective use of existing knowledge about the world and actions without retraining from scratch for each new task. Instead, WholeBodyWAM focuses on the structured alignment and grounding of this knowledge through whole-body controllers, significantly simplifying training and improving the robot’s ability to generalise.
Experimental Results and Prospects
Experiments using WholeBodyWAM produced impressive results. The overall success rate in simulation tasks was 91,9%. In real-world settings, performance on tasks outside the training data distribution (out-of-distribution) improved by 0,23 compared with baseline models. In addition, the variation in success rates across different whole-body controllers fell significantly—by 70% compared with baseline methods.
The findings indicate that WholeBodyWAM paves the way for scalable artificial intelligence systems for whole-body control of humanoid robots. Extending pre-trained models of world interactions through structured alignment with whole-body controllers is a promising direction for robotics development.
Source: cs.RO updates on arXiv.org
