BigBite思维随笔 . Big Bite Small Talk, 杂谈随笔，聊科技，AI，成长，理财，经验杂谈。Stay Hungry 作者 | BigBite 来源 | BigBite思维随笔 原文链接： 模仿学习无法真正端到端 点击下方 卡片 ，关注" 自动驾驶之心 "公众号 戳我-> 领取 自动驾驶近30个 方向 学习 路线 以下文章来源于BigBite思维随笔 ，作者BigBite >>自动驾驶前沿信息获取 → 自动驾驶之心知识星球 本文只做学术分享，如有侵权，联系删文 自动驾驶行业新的技术名词层出不穷，在大家争论到底是VLA更好，还是世界模型更先进的时候，其实忽略了相比模型架构，训练方法才是决定功能效 果的关键。事实上无论是VLA也好，世界行为模型也罢，本质上他们都是实现端到端的具体模型结构，可是随着越来越多头部企业在端到端的技术范式 上努力探索投入，头部团队逐渐发现单纯依靠模仿学习实现不了彻底的端到端自动驾驶！ 那么模仿学习在自动驾驶领域中的问题和局限性到底在哪里呢？ 模仿学习假定专家数据是最优的 模仿学习的潜在假设是每一条训练数据轨迹都给出了在当前状态下最优的行为真值，因此越接近训练数据的行 ...

Core Insights - The article discusses the evolution of end-to-end autonomous driving technology, highlighting the shift from horizontal to vertical end-to-end systems as a new industry focus [2][3] - It emphasizes the importance of vertical end-to-end control for achieving human-like driving efficiency, particularly in speed and braking control [4][16] Group 1: Importance of Vertical End-to-End Control - Vertical end-to-end control is essential for achieving smooth acceleration and deceleration, which is a key differentiator between novice and experienced drivers [3][4] - The article defines "defensive deceleration" as the ability to adjust speed based on necessity and prediction, balancing safety and efficiency [4][12] - Current autonomous systems often prioritize navigation efficiency over vertical control, making it challenging to implement effective speed adjustments [15][16] Group 2: Challenges in Achieving Vertical End-to-End Control - Many autonomous driving systems have successfully implemented horizontal end-to-end control, but vertical control remains a significant challenge [13][16] - The noise in human driving data complicates the learning process for autonomous systems, making it difficult to distinguish meaningful speed control from random fluctuations [16][17] - Solutions to improve vertical control include data cleaning, causal reasoning, and reinforcement learning, which are being explored by leading autonomous driving teams [17]

Core Viewpoint - The automotive industry is undergoing a significant transformation, with numerous executive changes and a focus on advanced technologies such as autonomous driving and artificial intelligence [1][3]. Group 1: Industry Changes - In September, 48 executives in the automotive sector underwent changes, indicating a shift in leadership and strategy [1]. - Companies like Li Auto and BYD are restructuring their teams to enhance their capabilities in autonomous driving and cockpit technology [1]. - The industry is witnessing a rapid evolution in algorithm development, moving from BEV to more complex models like VLA and world models [1][3]. Group 2: Autonomous Driving Focus - The forefront of autonomous driving technology is centered on VLA/VLM, end-to-end driving, world models, and reinforcement learning [3]. - There is a notable gap in understanding the industry's actual progress among students and mid-sized companies, highlighting the need for better communication between academia and industry [3]. Group 3: Community and Knowledge Sharing - A community called "Autonomous Driving Heart Knowledge Planet" has been established to bridge the gap between academic and industrial knowledge, aiming to grow to nearly 10,000 members in two years [5]. - The community offers a comprehensive platform for learning, including video content, Q&A, and job exchange, catering to both beginners and advanced learners [6][10]. - Members can access over 40 technical routes and engage with industry leaders to discuss trends and challenges in autonomous driving [6][8]. Group 4: Learning Resources - The community provides various resources for practical questions related to autonomous driving, such as entry points for end-to-end systems and data annotation practices [6][11]. - A detailed curriculum is available for newcomers, covering essential topics in autonomous driving technology [20][21]. - The platform also includes job referral mechanisms to connect members with potential employers in the autonomous driving sector [13][14].

Core Viewpoint - 2023 is identified as the year of end-to-end production, with 2024 expected to be a significant year for this development in the automotive industry, particularly in autonomous driving technology [1][3]. Group 1: End-to-End Production - Leading new forces and manufacturers have already achieved end-to-end production [1]. - There are two main paradigms in the industry: one-stage and two-stage approaches, with UniAD being a representative of the one-stage method [1]. Group 2: Development Trends - Since last year, the one-stage end-to-end approach has rapidly evolved, leading to various derivatives such as perception-based, world model-based, diffusion model-based, and VLA-based one-stage methods [3]. - Major autonomous driving companies are focusing on self-research and mass production of end-to-end autonomous driving solutions [3]. Group 3: Course Offerings - A course titled "End-to-End and VLA Autonomous Driving" has been launched, covering cutting-edge algorithms in both one-stage and two-stage end-to-end approaches [5]. - The course aims to provide insights into the latest technologies in the field, including BEV perception, visual language models, diffusion models, and reinforcement learning [5]. Group 4: Course Structure - The course consists of several chapters, starting with an introduction to end-to-end algorithms, followed by background knowledge essential for understanding the technology stack [9][10]. - The second chapter focuses on the most frequently asked technical keywords in job interviews over the next two years [10]. - Subsequent chapters delve into two-stage end-to-end methods, one-stage end-to-end methods, and practical assignments involving RLHF fine-tuning [12][13]. Group 5: Learning Outcomes - Upon completion, participants are expected to reach a level equivalent to one year of experience as an end-to-end autonomous driving algorithm engineer [19]. - The course aims to deepen understanding of key technologies such as BEV perception, multimodal large models, and reinforcement learning, enabling participants to apply learned concepts to real projects [19].

Core Viewpoint - ReflectDrive is a novel learning framework that integrates a reflective mechanism to achieve safe trajectory generation through discrete diffusion, addressing the challenges in end-to-end autonomous driving systems [4][46]. Group 1: Introduction and Background - Autonomous driving is leading the transportation industry towards a safer and more efficient future, with end-to-end (E2E) systems becoming a mainstream alternative to traditional modular designs [4]. - Visual-Language-Action (VLA) models combine pre-trained knowledge from visual-language models (VLM) to enhance adaptability in complex scenarios [4][5]. - Current learning-based methods have not resolved core challenges in imitation learning driving systems, particularly in encoding physical rules like collision avoidance [4][5]. Group 2: ReflectDrive Framework - ReflectDrive proposes a new learning framework that utilizes a discrete diffusion reflective mechanism for safe trajectory generation [3][12]. - The framework begins by discretizing the two-dimensional driving space to construct an action codebook, allowing fine-tuning of pre-trained diffusion language models for planning tasks [3][14]. - The reflective mechanism operates without gradient calculations, enabling iterative self-correction inspired by spatiotemporal joint planning [3][8]. Group 3: Methodology and Mechanism - The reflective inference process consists of two stages: target condition trajectory generation and safety-guided regeneration [20][25]. - The framework integrates safety metrics to evaluate generated multimodal trajectories, identifying unsafe path points through local search methods [8][25]. - The iterative optimization loop continues until the trajectory is deemed safe or computational limits are reached, ensuring high efficiency in real-time performance [31][32]. Group 4: Experimental Results - ReflectDrive was evaluated on the NAVSIM benchmark, demonstrating significant improvements in safety metrics such as collision rates and compliance with drivable areas [32][38]. - The introduction of the safety-guided regeneration mechanism led to substantial enhancements in safety indicators, with notable increases in DAC (3.9%), TTC (1.3%), NC (0.8%), and EP (7.9%) compared to the baseline [37][38]. - When using ground-truth agent information, ReflectDrive's performance approached human driving levels, achieving NC of 99.7% and DAC of 99.5% [38][39]. Group 5: Conclusion - ReflectDrive effectively integrates a reflective mechanism with discrete diffusion for safe trajectory generation, validated by its performance on the NAVSIM benchmark [46].

Core Insights - The article discusses the competition between VLA (Vision-Language Action) models and world models in the field of end-to-end autonomous driving, highlighting that over 90% of current models are segmented end-to-end rather than purely VLA or world models [2][6]. Group 1: Model Comparison - VLA models, represented by companies like Gaode Map and Horizon Robotics, show superior performance compared to world models, with the latest VLA papers published in September 2023 [6][43]. - The performance metrics of various models indicate that VLA models outperform world models significantly, with the best VLA model achieving an average L2 distance of 0.19 meters and a collision rate of 0.08% [5][6]. Group 2: Data Utilization - The Shanghai AI Lab's GenAD model utilizes unlabelled data sourced from the internet, primarily YouTube, to enhance generalization capabilities, contrasting with traditional supervised learning methods that rely on labeled data [7][19]. - The GenAD framework employs a two-tier training approach similar to Tesla's, integrating diffusion models and Transformers, but requires high-precision maps and traffic rules for effective operation [26][32]. Group 3: Testing Methods - Two primary testing methods for end-to-end autonomous driving are identified: open-loop testing using synthetic data in simulators like CARLA, and closed-loop testing based on real-world collected data [4][6]. - The article emphasizes the limitations of open-loop testing, which cannot provide feedback on the execution of predicted actions, making closed-loop testing more reliable for evaluating model performance [4][6]. Group 4: Future Directions - The article suggests that while world models have potential, their current implementations often require additional labeled data, which diminishes their advantages in generalization and cost-effectiveness compared to VLA models [43]. - The ongoing research and development in the field indicate a trend towards improving the integration of various data sources and enhancing model robustness through advanced training techniques [19][32].

Core Insights - The article introduces AnchDrive, an end-to-end framework for autonomous driving that effectively addresses the challenges of multimodal behavior and generalization in long-tail scenarios [1][10][38] - AnchDrive utilizes a hybrid trajectory anchor approach, combining dynamic and static anchors to enhance trajectory quality and robustness in planning [10][38] Group 1: Introduction and Background - End-to-end autonomous driving algorithms have gained significant attention due to their superior scalability and adaptability compared to traditional rule-based motion planning methods [4][12] - These methods learn control signals directly from raw sensor data, reducing the complexity of modular design and minimizing cumulative perception errors [4][12] Group 2: Methodology - AnchDrive employs a multi-head trajectory decoder that dynamically generates a set of trajectory anchors, capturing behavioral diversity under local environmental conditions [8][15] - The framework integrates a large-scale static anchor set derived from human driving data, providing cross-scenario behavioral prior knowledge [8][15] Group 3: Experimental Results - In the NAVSIM v2 simulation platform, AnchDrive achieved an Extended Predictive Driver Model Score (EPDMS) of 85.5, indicating its ability to generate robust and contextually appropriate behaviors in complex driving scenarios [9][30][34] - The performance of AnchDrive was significantly higher than existing methods, with an 8.9 point increase in EPDMS compared to VADv2, while reducing the number of trajectory anchors from 8192 to just 20 [34] Group 4: Contributions - The main contributions of the article include the introduction of the AnchDrive framework, which utilizes a truncated diffusion process initialized from a hybrid trajectory anchor set, significantly improving initial trajectory quality and planning robustness [10][38] - The design of a mixed perception model with dense and sparse branches enhances the planner's understanding of obstacles and road geometry [11][18]

Core Insights - The article discusses the limitations of current autonomous driving systems in achieving true scene understanding and proposes a new framework called OmniScene, which integrates human-like cognitive abilities into the driving process [11][13][14]. Group 1: OmniScene Framework - OmniScene introduces a visual-language model (OmniVLM) that combines panoramic perception with temporal fusion capabilities for comprehensive 4D scene understanding [2][14]. - The framework employs a teacher-student architecture for knowledge distillation, embedding textual representations into 3D instance features to enhance semantic supervision [2][15]. - A hierarchical fusion strategy (HFS) is proposed to address the imbalance in contributions from different modalities during multi-modal fusion, allowing for adaptive calibration of geometric and semantic features [2][16]. Group 2: Performance Evaluation - OmniScene was evaluated on the nuScenes dataset, outperforming over ten mainstream models across various tasks, establishing new benchmarks for perception, prediction, planning, and visual question answering (VQA) [3][16]. - Notably, OmniScene achieved a significant 21.40% improvement in visual question answering performance, demonstrating its robust multi-modal reasoning capabilities [3][16]. Group 3: Human-like Scene Understanding - The framework aims to replicate human visual processing by continuously converting sensory input into scene understanding, adjusting attention based on dynamic driving environments [11][14]. - OmniVLM is designed to process multi-view and multi-frame visual inputs, enabling comprehensive scene perception and attention reasoning [14][15]. Group 4: Multi-modal Learning - The proposed HFS combines 3D instance representations with multi-view visual inputs and semantic attention derived from textual cues, enhancing the model's ability to understand complex driving scenarios [16][19]. - The integration of visual and textual modalities aims to improve the model's contextual awareness and decision-making processes in dynamic environments [19][20]. Group 5: Challenges and Solutions - The article highlights challenges in integrating visual-language models (VLMs) into autonomous driving, such as the need for domain-specific knowledge and real-time safety requirements [20][21]. - Solutions include designing driving attention prompts and developing new end-to-end visual-language reasoning methods to address safety-critical driving scenarios [22].

Core Insights - The article introduces FlowDrive, a novel end-to-end driving framework that integrates energy-based flow field representation, adaptive anchor trajectory optimization, and motion-decoupled trajectory generation to enhance safety and interpretability in autonomous driving [4][45]. Group 1: Introduction and Background - End-to-end autonomous driving has gained attention for its potential to simplify traditional modular pipelines and leverage large-scale data for joint learning of perception, prediction, and planning tasks [4]. - A mainstream research direction involves generating Bird's Eye View (BEV) representations from multi-view camera inputs, which provide structured spatial views beneficial for downstream planning tasks [4][6]. Group 2: FlowDrive Framework - FlowDrive introduces energy-based flow fields in the BEV space to explicitly model geometric constraints and rule-based semantics, enhancing the effectiveness of BEV representations [7][15]. - The framework includes a flow-aware anchor trajectory optimization module that aligns initial trajectories with safe and goal-oriented areas, improving spatial effectiveness and intention consistency [15][22]. - A task-decoupled diffusion planner separates high-level intention prediction from low-level trajectory denoising, allowing for targeted supervision and flow field conditional decoding [9][27]. Group 3: Experimental Results - Experiments on the NAVSIM v2 benchmark dataset demonstrate that FlowDrive achieves state-of-the-art performance, with an Extended Predictive Driver Model Score (EPDMS) of 86.3, surpassing previous benchmark methods [3][40]. - FlowDrive shows significant advantages in safety-related metrics such as Drivable Area Compliance (DAC) and Time to Collision (TTC), indicating superior adherence to driving constraints and hazard avoidance capabilities [40][41]. - The framework's performance is validated through ablation studies, showing that removing any core component leads to significant declines in overall performance [43][47]. Group 4: Technical Details - The flow field learning module encodes dense, physically interpretable spatial gradients to provide fine-grained guidance for trajectory planning [20][21]. - The perception module utilizes a Transformer-based architecture to effectively fuse multi-modal sensor inputs into a compact and semantically rich BEV representation [18][37]. - The training process involves a composite loss function that supervises trajectory planning, anchor trajectory optimization, flow field modeling, and auxiliary perception tasks [30][31][32][34].

Core Viewpoint - The article discusses the evolution of Bird's Eye View (BEV) perception as a foundational technology for autonomous driving, highlighting its importance in ensuring safety and reliability in complex driving environments [2][4]. Group 1: Essence of BEV Perception - BEV perception is an efficient spatial representation paradigm that projects heterogeneous data from various sensors (like cameras, LiDAR, and radar) into a unified BEV coordinate system, facilitating a consistent structured spatial semantic map [6][12]. - This top-down view significantly reduces the complexity of multi-view and multi-modal data fusion, aiding in the accurate perception and understanding of spatial relationships between objects [6][12]. Group 2: Importance of BEV Perception - With a unified and interpretable spatial representation, BEV perception serves as an ideal foundation for multi-modal fusion and multi-agent collaborative perception in autonomous driving [8][12]. - The integration of heterogeneous sensor data into a common BEV plane allows for seamless alignment and integration, enhancing the efficiency of information sharing between vehicles and infrastructure [8][12]. Group 3: Implementation of BEV Perception - The evolution of safety-oriented BEV perception (SafeBEV) is categorized into three main stages: SafeBEV 1.0 (single-modal vehicle perception), SafeBEV 2.0 (multi-modal vehicle perception), and SafeBEV 3.0 (multi-agent collaborative perception) [12][17]. - Each stage represents advancements in technology and features, addressing the increasing complexity of dynamic traffic scenarios [12][17]. Group 4: SafeBEV 1.0 - Single-Modal Vehicle Perception - This stage utilizes a single sensor (like a camera or LiDAR) for BEV scene understanding, with methods evolving from homography transformations to data-driven BEV modeling [13][19]. - The performance of camera-based methods is sensitive to lighting changes and occlusions, while LiDAR methods face challenges with point cloud sparsity and performance degradation in adverse weather [19][41]. Group 5: SafeBEV 2.0 - Multi-Modal Vehicle Perception - Multi-modal BEV perception integrates data from cameras, LiDAR, and radar to enhance performance and robustness in challenging conditions [42][45]. - Fusion strategies are categorized into five types, including camera-radar, camera-LiDAR, radar-LiDAR, camera-LiDAR-radar, and temporal fusion, each leveraging the complementary characteristics of different sensors [42][45]. Group 6: SafeBEV 3.0 - Multi-Agent Collaborative Perception - The development of Vehicle-to-Everything (V2X) technology enables autonomous vehicles to exchange information and perform joint reasoning, overcoming the limitations of single-agent perception [15][16]. - Collaborative perception aggregates multi-source sensor data in a unified BEV space, facilitating global environmental modeling and enhancing safety navigation in dynamic traffic [15][16]. Group 7: Challenges and Future Directions - The article identifies key challenges in open-world scenarios, such as open-set recognition, large-scale unlabeled data, sensor performance degradation, and communication delays among agents [17]. - Future research directions include the integration of BEV perception with end-to-end autonomous driving systems, embodied intelligence, and large language models [17].