WorldVLA：世界模型实现视觉-动作双向增强，抓取精度显著提升

Core Insights - The article introduces WorldVLA, a self-regressive action world model that integrates action and image understanding and generation, outperforming independent action and world models [3][6][8]. Group 1: WorldVLA Overview - WorldVLA combines visual-language-action (VLA) models and world models in a single framework, enhancing performance through mutual reinforcement between the two components [3][6]. - The model utilizes three independent tokenizers for images, text, and actions, sharing the same vocabulary to unify cross-modal understanding and generation [6][14]. - An attention mask strategy is proposed to mitigate error propagation in action sequence generation, significantly improving performance in action block generation tasks [7][31]. Group 2: Model Architecture and Training - The architecture consists of an action model and a world model, where the action model generates actions based on image observations and language instructions, while the world model predicts future states based on observed sequences and actions [11][13]. - Training involves mixing action model data and world model data to enhance action generation, with the world model providing a better understanding of environmental physics [15][20]. - The loss function combines cross-entropy losses from both models, balancing contributions due to the disparity in token counts [20]. Group 3: Experimental Results - WorldVLA shows a 4% higher success rate in grasping tasks compared to similar action models and a 10% reduction in Fréchet Video Distance (FVD) compared to standard world models [7][26]. - The model's performance improves with higher image resolutions, which is crucial for tasks requiring high operational precision [26]. - The integration of the world model significantly enhances the action model's performance by providing a better understanding of the underlying physical dynamics [28]. Group 4: Attention Mask and Performance - The proposed attention mask allows for parallel generation of multiple actions, reducing dependency on previous actions and alleviating error accumulation [19][31]. - The model's performance is optimized by using two historical image frames as input, balancing task success rates and computational efficiency [32]. Group 5: Pre-training and Future Potential - Pre-training the action model with world model data significantly improves grasping performance, highlighting the potential of leveraging general world knowledge to enhance specific task performance in robotics [35].