ICCV 2025 | HERMES：首个统一3D场景理解与生成的世界模型

Core Viewpoint - The article discusses the advancements in autonomous driving technology, emphasizing the need for a unified model that integrates both understanding current environments and predicting future scenarios effectively [7][10][30]. Research Background and Motivation - Recent progress in autonomous driving necessitates vehicles to possess deep understanding of current environments and accurate predictions of future scenarios to ensure safe and efficient navigation [7]. - The separation of "understanding" and "generation" in mainstream solutions is highlighted as a limitation in achieving effective decision-making in real-world driving scenarios [8][10]. Method: HERMES Unified Framework - HERMES proposes a unified framework that utilizes a shared large language model (LLM) to drive both understanding and generation tasks simultaneously [13][30]. - The framework addresses challenges such as efficiently inputting high-resolution images and integrating world knowledge with predictive capabilities [11][12]. HERMES Core Design - HERMES employs Bird's-Eye View (BEV) as a unified scene representation, allowing for efficient encoding of multiple images while preserving spatial relationships and semantic details [18]. - The introduction of World Queries facilitates the connection between understanding and future predictions, enhancing the model's ability to generate accurate future scenarios [19][20]. Joint Training and Optimization - HERMES utilizes a joint training process with two optimization objectives: language modeling loss for understanding tasks and point cloud generation loss for accuracy in future predictions [21][22][23]. Experimental Results and Visualization - HERMES demonstrates superior performance in scene understanding and future generation tasks on datasets like nuScenes and OmniDrive-nuScenes [26]. - The model excels in generating coherent future point clouds and accurately describing driving scenes, showcasing its comprehensive capabilities [27]. Summary and Future Outlook - HERMES presents a new paradigm for autonomous driving world models, effectively bridging the gap between 3D scene understanding and future generation [30]. - The model shows significant improvements in prediction accuracy and understanding tasks compared to traditional models, validating the effectiveness of unified modeling [31].