在端到端自动驾驶领域，这篇文章是一个典型的"两段式网络架构"中的Planner模型，而且它不是基于BEV feature map进行下游控制任务的，而是直接对于感知输出 的结构化的信息（bbox，lanes等等）进行编码，并作为sequence token输入到decoder中，今天就为大家分享一下。二段式端到端非常适合新人练手： 为了帮助大家理解，网络架构图上我们做了详细的模块注释： 我们先整体上看一下PLUTO有哪些关键点： PLUTO主要有三个损失，主任务的损失包含回归损失和分类损失，共同组成模仿学习的损失。而Agent轨迹预测的损失如下图所示： 同时，PLUTO也添加了几个辅助的损失帮助模型收敛： 1）直击痛点，快速入门 本课程基于Just-in-Time Learning理念，通过通俗易懂的语言和案例，帮助学员短时间内掌握核心技术栈。理解关键概念后，拓展特定领域知识将变得更加轻松。 2）构建领域框架，提升研究能力 本文均出自平台最新推出的 『端到端与VLA自动驾驶小班课』 ，我们联合国内TOP主机厂算法专家共同打造！ 技术专家带你深入端到端与VLA算法原理与技术开 发，目前已经正式开课！ 技术栈多？ ...

作者 | 钱红中 编辑 | 自动驾驶之心 原文链接： https://zhuanlan.zhihu.com/p/12088531309 点击下方 卡片 ，关注" 自动驾驶之心 "公众号 戳我-> 领取 自动驾驶近30个 方向 学习 路线 >>自动驾驶前沿信息获取 → 自动驾驶之心知识星球 本文只做学术分享，如有侵权，联系删文 Q：端到端模型通常大致分为几种？ 分为两种，一种是完全黑盒OneNet，模型直接优化Planner；另一种是模块化端到端，即模块级联或者并联，通过感知模块，预测模块以及规划模块之间feat- level/query-level的交互，减少分段式自动驾驶模型的误差累积。 Q：[UniAD] 整个框架分为4部分，输入multi-view camera imgs，Backbone模块提取BEV feat，Perception模块完成对于scene-level的感知包括对于agents+ego以及map， Prediction模块基于时序交互以及agents-scene的交互完成对于agents+ego的multi-mode轨迹预测，Planner模块基于预测的轨迹以及BEV feat完成路径的 ...

Core Viewpoint - The article discusses the rapid evolution and challenges in the field of end-to-end autonomous driving technology, emphasizing the need for a comprehensive understanding of various algorithms and models to succeed in this competitive industry [2][4][6]. Group 1: Industry Trends - The shift from modular approaches to end-to-end systems in autonomous driving aims to eliminate cumulative errors between modules, marking a significant technological leap [2]. - The emergence of various algorithms and models, such as UniAD and BEV perception, indicates a growing focus on integrating multiple tasks into a unified framework [4][9]. - The demand for knowledge in multi-modal large models, reinforcement learning, and diffusion models is increasing, reflecting the industry's need for versatile skill sets [5][20]. Group 2: Learning Challenges - New entrants face difficulties due to the fragmented nature of knowledge and the overwhelming volume of research papers in the field, often leading to early abandonment of learning [5][6]. - The lack of high-quality documentation and practical guidance further complicates the transition from theory to practice in end-to-end autonomous driving research [5][6]. Group 3: Course Offerings - A new course titled "End-to-End and VLA Autonomous Driving" has been developed to address the learning challenges, focusing on practical applications and theoretical foundations [6][24]. - The course is structured to provide a comprehensive understanding of end-to-end algorithms, including their historical development and current trends [11][12]. - Practical components, such as real-world projects and assignments, are included to ensure that participants can apply their knowledge effectively [8][21]. Group 4: Course Content Overview - The course covers various topics, including the introduction to end-to-end algorithms, background knowledge on relevant technologies, and detailed explorations of both one-stage and two-stage end-to-end methods [11][12][13]. - Specific chapters focus on advanced topics like world models and diffusion models, which are crucial for understanding the latest advancements in autonomous driving [15][17][20]. - The final project involves practical applications of reinforcement learning from human feedback (RLHF), allowing participants to gain hands-on experience [21].

Core Viewpoint - The article discusses the development and significance of end-to-end (E2E) algorithms in autonomous driving, emphasizing the integration of various advanced technologies such as large language models (LLMs), diffusion models, and reinforcement learning (RL) in enhancing the capabilities of autonomous systems [21][31]. Summary by Sections Section 1: Overview of End-to-End Autonomous Driving - The first chapter provides a comprehensive overview of the evolution of end-to-end algorithms, explaining the transition from modular approaches to end-to-end solutions, and discussing the advantages and challenges of different paradigms [40]. Section 2: Background Knowledge - The second chapter focuses on the technical stack associated with end-to-end systems, detailing the importance of LLMs, diffusion models, and reinforcement learning, which are crucial for understanding the future job market in this field [41][42]. Section 3: Two-Stage End-to-End Systems - The third chapter delves into two-stage end-to-end systems, exploring their emergence, advantages, and disadvantages, while also reviewing notable works in the field such as PLUTO and CarPlanner [42][43]. Section 4: One-Stage End-to-End and VLA - The fourth chapter highlights one-stage end-to-end systems, discussing various subfields including perception-based methods and the latest advancements in VLA (Vision-Language Alignment), which are pivotal for achieving the ultimate goals of autonomous driving [44][50]. Section 5: Practical Application and RLHF Fine-Tuning - The fifth chapter includes a major project focused on RLHF (Reinforcement Learning from Human Feedback) fine-tuning, providing practical insights into building pre-training and reinforcement learning modules, which are applicable to VLA-related algorithms [52]. Course Structure and Learning Outcomes - The course aims to equip participants with a solid understanding of end-to-end autonomous driving technologies, covering essential frameworks and methodologies, and preparing them for roles in the industry [56][57].

Core Viewpoint - The article announces the launch of the "Heart of Autonomous Driving" project and paper guidance, aimed at assisting students facing challenges in their research and development efforts in the field of autonomous driving [1]. Group 1: Project and Guidance Overview - The project aims to provide support for students who encounter difficulties in their research, such as environmental configuration issues and debugging challenges [1]. - Last year's outcomes were positive, with several students successfully publishing papers in top conferences like CVPR and ICRA [1]. Group 2: Guidance Directions - **Direction 1**: Focus on multi-modal perception and computer vision, end-to-end autonomous driving, large models, and BEV perception. The guiding teacher has published over 30 papers in top AI conferences with a citation count exceeding 6000 [3]. - **Direction 2**: Emphasis on 3D Object Detection, Semantic Segmentation, Occupancy Prediction, and multi-task learning based on images or point clouds. The guiding teacher is a top-tier PhD with multiple publications in ECCV and CVPR [5]. - **Direction 3**: Concentration on end-to-end autonomous driving, OCC, BEV, and world model directions. The guiding teacher is also a top-tier PhD with contributions to several mainstream perception solutions [6]. - **Direction 4**: Focus on NeRF / 3D GS neural rendering and 3D reconstruction. The guiding teacher has published four CCF-A class papers, including two in CVPR and two in IEEE Transactions [7].

Core Viewpoint - End-to-End Autonomous Driving is a key algorithm for intelligent driving mass production, with significant salary potential for related positions, and it has evolved into various technical directions since the introduction of UniAD [2][4]. Group 1: Technical Directions - End-to-End Autonomous Driving can be categorized into one-stage and two-stage approaches, with various subfields emerging under each category [2][4]. - The core advantage of end-to-end systems is the direct modeling from sensor input to vehicle planning/control information, avoiding error accumulation seen in modular methods [2]. - Notable algorithms include PLUTO for two-stage end-to-end, UniAD for perception-based one-stage, OccWorld for world model-based one-stage, and DiffusionDrive for diffusion model-based one-stage [4]. Group 2: Industry Trends - The demand for VLA/VLM algorithm experts is increasing, with salary ranges for positions requiring 3-5 years of experience being between 40K-70K [9]. - The industry is witnessing a shift towards large model algorithms, with companies focusing on VLA as the next generation of autonomous driving solutions [8][9]. Group 3: Course Offerings - A new course titled "End-to-End and VLA Autonomous Driving" is being offered to help individuals understand the complexities of end-to-end algorithms and their applications [15][28]. - The course covers various topics, including background knowledge, two-stage end-to-end, one-stage end-to-end, and practical applications of reinforcement learning [20][22][24]. - The course aims to provide a comprehensive understanding of the end-to-end framework, including key technologies like BEV perception, multi-modal large models, and diffusion models [31].

Core Viewpoint - End-to-End Autonomous Driving is a key algorithm for intelligent driving mass production, with significant salary potential for related positions, and it has evolved into various technical branches since the introduction of UniAD [2] Group 1: Overview of End-to-End Autonomous Driving - End-to-End Autonomous Driving can be categorized into one-stage and two-stage approaches, with the core advantage being direct modeling from sensor input to vehicle planning/control, avoiding error accumulation seen in modular methods [2] - The emergence of BEV perception has bridged gaps between modular methods, leading to a significant technological leap [2] - The academic and industrial focus on End-to-End technology has raised questions about whether UniAD is the ultimate solution, indicating ongoing developments in various algorithms [2] Group 2: Challenges in Learning - The rapid development of End-to-End technology has made previous solutions inadequate, necessitating knowledge in multimodal large models, BEV perception, reinforcement learning, visual transformers, and diffusion models [4] - Beginners often struggle with the fragmented nature of knowledge and the overwhelming number of papers, leading to challenges in extracting frameworks and understanding industry trends [4] Group 3: Course Features - The newly developed course on End-to-End and VLA Autonomous Driving aims to address learning challenges by providing a structured approach to mastering core technologies [5] - The course emphasizes Just-in-Time Learning, helping students quickly grasp key concepts and expand their knowledge in specific areas [5] - It aims to build a framework for research capabilities, enabling students to categorize papers and extract innovative points [6] Group 4: Course Outline - The course includes chapters on the introduction to End-to-End algorithms, background knowledge, two-stage End-to-End methods, one-stage End-to-End methods, and practical applications [11][12][13] - Key topics include the evolution of End-to-End methods, the significance of BEV perception, and the latest advancements in VLA [9][14] Group 5: Target Audience and Expected Outcomes - The course is designed for individuals aiming to enter the autonomous driving industry, providing a comprehensive understanding of End-to-End technologies [19] - Upon completion, participants are expected to achieve a level equivalent to one year of experience as an End-to-End Autonomous Driving algorithm engineer, mastering various methodologies and key technologies [22]

Core Viewpoint - End-to-End Autonomous Driving (E2E) is the core algorithm for intelligent driving mass production, marking a new phase in the industry with significant advancements and competition following the recognition of UniAD at CVPR [2] Group 1: E2E Autonomous Driving Overview - E2E can be categorized into single-stage and two-stage approaches, directly modeling from sensor data to vehicle control information, thus avoiding error accumulation seen in modular methods [2] - The emergence of BEV perception has bridged gaps between modular methods, leading to a significant technological leap [2] - The rapid development of E2E has led to a surge in demand for VLM/VLA expertise, with potential salaries reaching millions annually [2] Group 2: Learning Challenges - The fast-paced evolution of E2E technology has made previous learning materials outdated, necessitating a comprehensive understanding of multi-modal large models, BEV perception, reinforcement learning, and more [3] - Beginners face challenges in synthesizing knowledge from numerous fragmented papers and transitioning from theory to practice due to a lack of high-quality documentation [3] Group 3: Course Development - A new course titled "End-to-End and VLA Autonomous Driving" has been developed to address learning challenges, focusing on Just-in-Time Learning to help students quickly grasp core technologies [4] - The course aims to build a framework for research capabilities, enabling students to categorize papers and extract innovative points [5] - Practical applications are integrated into the course to ensure a complete learning loop from theory to practice [6] Group 4: Course Structure - The course consists of multiple chapters covering the history and evolution of E2E algorithms, background knowledge, two-stage and one-stage E2E methods, and the latest advancements in VLA [8][9][10] - Key topics include the introduction of E2E algorithms, background knowledge on VLA, and practical applications of diffusion models and reinforcement learning [11][12] Group 5: Target Audience and Outcomes - The course is designed for individuals with a foundational understanding of autonomous driving and aims to elevate participants to a level comparable to one year of experience as an E2E algorithm engineer [19] - Participants will gain a deep understanding of key technologies such as BEV perception, multi-modal large models, and reinforcement learning, enabling them to apply learned concepts to real-world projects [19]