Workflow
ORION
icon
Search documents
清华教研团队!两个月从零搭建一套自己的自动驾驶VLA模型
自动驾驶之心· 2025-09-28 07:21
Core Viewpoint - The focus of academia and industry after end-to-end systems is on VLA (Vision-Language-Action), which provides human-like reasoning capabilities for safer and more reliable autonomous driving [1][4]. Summary by Sections Introduction to Autonomous Driving VLA - VLA is categorized into modular VLA, integrated VLA, and reasoning-enhanced VLA, which are essential for advancing autonomous driving technology [1][4]. Technical Maturity and Employment Demand - The demand for autonomous driving VLA solutions is high among major companies, prompting them to invest in self-research and development [4]. Course Overview - A comprehensive learning roadmap for autonomous driving VLA has been designed, covering principles to practical applications [4][6]. Core Content of Autonomous Driving VLA - Key topics include visual perception, large language models, action modeling, model deployment, and dataset creation, with cutting-edge algorithms like CoT, MoE, RAG, and reinforcement learning [6]. Course Collaboration - The course is developed in collaboration with Tsinghua University's research team, featuring detailed explanations of algorithms and practical assignments [6]. Course Structure - The course consists of six chapters, each focusing on different aspects of VLA, including algorithm introduction, foundational algorithms, VLM as an interpreter, modular and integrated VLA, reasoning-enhanced VLA, and a final project [12][20]. Chapter Details - Chapter 1 covers the concept and history of VLA algorithms, including benchmarks and evaluation metrics [13]. - Chapter 2 focuses on foundational algorithms related to Vision, Language, and Action, along with model deployment [14]. - Chapter 3 discusses VLM's role as an interpreter in autonomous driving, highlighting key algorithms [15]. - Chapter 4 delves into modular and integrated VLA, emphasizing the evolution of language models in planning [16]. - Chapter 5 explores reasoning-enhanced VLA, introducing new modules for decision-making and action output [17]. - Chapter 6 involves a hands-on project where participants build and fine-tune their models [20]. Learning Outcomes - The course aims to deepen understanding of VLA's current advancements and core algorithms, equipping participants with practical skills for future research and applications in the autonomous driving sector [22][26]. Course Schedule - The course is set to begin on October 20, with a structured timeline for each chapter's release [23]. Prerequisites - Participants are expected to have a foundational knowledge of autonomous driving, large models, reinforcement learning, and programming skills in Python and PyTorch [26].
论文解读之港科PLUTO:首次超越Rule-Based的规划器!
自动驾驶之心· 2025-09-15 23:33
Core Viewpoint - The article discusses the development and features of the PLUTO model within the end-to-end autonomous driving domain, emphasizing its unique two-stage architecture and its direct encoding of structured perception outputs for downstream control tasks [1][2]. Summary by Sections Overview of PLUTO - PLUTO is characterized by its three main losses: regression loss, classification loss, and imitation learning loss, which collectively contribute to the model's performance [7]. - Additional auxiliary losses are incorporated to aid model convergence [9]. Course Introduction - The article introduces a new course titled "End-to-End and VLA Autonomous Driving," developed in collaboration with top algorithm experts from domestic leading manufacturers, aimed at addressing the challenges faced by learners in this rapidly evolving field [12][15]. Learning Challenges - The course addresses the difficulties learners face due to the fast-paced development of technology and the fragmented nature of knowledge across various domains, making it hard for beginners to grasp the necessary concepts [13]. Course Features - The course is designed to provide quick entry into the field, build a framework for research capabilities, and combine theory with practical applications [15][16][17]. Course Outline - The course consists of several chapters covering topics such as the history and evolution of end-to-end algorithms, background knowledge on various technologies, and detailed discussions on both one-stage and two-stage end-to-end methods [20][21][22][29]. Practical Application - The course includes practical assignments, such as RLHF fine-tuning, allowing students to apply their theoretical knowledge in real-world scenarios [31]. Instructor Background - The instructor, Jason, has a strong academic and practical background in cutting-edge algorithms related to end-to-end and large models, contributing to the course's credibility [32]. Target Audience and Expected Outcomes - The course is aimed at individuals with a foundational understanding of autonomous driving and related technologies, with the goal of elevating their skills to the level of an end-to-end autonomous driving algorithm engineer within a year [36].
闭环端到端暴涨20%!华科&小米打造开源框架ORION
自动驾驶之心· 2025-08-30 16:03
Core Viewpoint - The article discusses the advancements in end-to-end (E2E) autonomous driving technology, particularly focusing on the introduction of the ORION framework, which integrates vision-language models (VLM) for improved decision-making in complex environments [3][30]. Summary by Sections Introduction - Recent progress in E2E autonomous driving technology faces challenges in complex closed-loop interactions due to limited causal reasoning capabilities [3][12]. - VLMs offer new hope for E2E autonomous driving but there remains a significant gap between VLM's semantic reasoning space and the numerical action space required for driving [3][17]. ORION Framework - ORION is proposed as an end-to-end autonomous driving framework that utilizes visual-language instructions for trajectory generation [3][18]. - The framework incorporates QT-Former for aggregating long-term historical context, VLM for scene understanding and reasoning, and a generative model to align reasoning and action spaces [3][16][18]. Performance Evaluation - ORION achieved a driving score of 77.74 and a success rate of 54.62% on the challenging Bench2Drive dataset, outperforming previous state-of-the-art (SOTA) methods by 14.28 points and 19.61% in success rate [5][24]. - The framework demonstrated superior performance in specific driving scenarios such as overtaking (71.11%), emergency braking (78.33%), and traffic sign recognition (69.15%) [26]. Key Contributions - The article highlights several key contributions of ORION: 1. QT-Former enhances the model's understanding of historical scenes by effectively aggregating long-term visual context [20]. 2. VLM enables multi-dimensional analysis of driving scenes, integrating user instructions and historical information for action reasoning [21]. 3. The generative model aligns the reasoning space of VLM with the action space for trajectory prediction, ensuring reasonable driving decisions in complex scenarios [22]. Conclusion - ORION provides a novel solution for E2E autonomous driving by achieving semantic and action space alignment, integrating long-term context aggregation, and jointly optimizing visual understanding and path planning tasks [30].
THEON announces new strategic US and European investments and partnerships to build global leadership in Digital and Augmented Reality defense optronics domain under the THEON NEXT initiative
GlobeNewswire News Room· 2025-08-11 23:12
Core Insights - Theon International Plc is launching the THEON NEXT initiative, focusing on strategic investments and partnerships to develop next-generation soldier systems through targeted collaborations and co-development efforts [1][2] Investment and Partnerships - THEON is making four significant investments and partnerships in the US and Europe, reinforcing its commitment to innovation and transatlantic cooperation in defense technologies [2][3] - A total investment of $15 million in Kopin Corporation includes a $7 million interest-bearing loan and an $8 million capital increase for a 49% stake in Kopin's Scottish subsidiary, aimed at co-developing AR-enabled systems [3][4] - A two-year renewable supply agreement has been signed with eMagin for OLED micro-displays, which are critical for THEON's products [7] - A strategic partnership with ALEREON will integrate Ultra-Wide-Band technology into THEON's A.R.M.E.D. product line, enhancing secure communication capabilities [8] - THEON is investing €5 million in Varjo Technologies Oy, a Finnish company specializing in VR and MR technologies, to support the development of high-tech products for defense applications [9] Technological Focus - The initiative emphasizes three critical technologies: Augmented and Virtual Reality software, micro-displays, and near-range wireless connectivity, which are essential for next-generation soldier systems [5] - THEON's A.R.M.E.D. product line is positioned to leverage these technologies, enhancing operational effectiveness in modern warfare [10] Strategic Goals - Theon aims to maintain its leadership in man-portable electro-optics while fostering US-European industrial cooperation [2][10] - The company plans to expand its operations into Germany and Belgium, establishing a thermal/digital hub in the EU [10] Leadership Statements - The CEO of THEON highlighted the importance of these partnerships in advancing the development of soldier-borne systems and enhancing operational capabilities [10] - The CFO noted that these strategic agreements represent a financially efficient investment approach, with a total investment of €25 million expected to yield quick returns [10]
自动驾驶端到端VLA落地,算法如何设计?
自动驾驶之心· 2025-06-22 14:09
Core Insights - The article discusses the rapid advancements in end-to-end autonomous driving, particularly focusing on Vision-Language-Action (VLA) models and their applications in the industry [2][3]. Group 1: VLA Model Developments - The introduction of AutoVLA, a new VLA model that integrates reasoning and action generation for end-to-end autonomous driving, shows promising results in semantic reasoning and trajectory planning [3][4]. - ReCogDrive, another VLA model, addresses performance issues in rare and long-tail scenarios by utilizing a three-stage training framework that combines visual language models with diffusion planners [7][9]. - Impromptu VLA introduces a dataset aimed at improving VLA models' performance in unstructured extreme conditions, demonstrating significant performance improvements in established benchmarks [14][24]. Group 2: Experimental Results - AutoVLA achieved competitive performance metrics in various scenarios, with the best-of-N method reaching a PDMS score of 92.12, indicating its effectiveness in planning and execution [5]. - ReCogDrive set a new state-of-the-art PDMS score of 89.6 on the NAVSIM benchmark, showcasing its robustness and safety in driving trajectories [9][10]. - The OpenDriveVLA model demonstrated superior results in open-loop trajectory planning and driving-related question-answering tasks, outperforming previous methods on the nuScenes dataset [28][32]. Group 3: Industry Trends - The article highlights a trend among major automotive manufacturers, such as Li Auto, Xiaomi, and XPeng, to invest heavily in VLA model research and development, indicating a competitive landscape in autonomous driving technology [2][3]. - The integration of large language models (LLMs) with VLA frameworks is becoming a focal point for enhancing decision-making capabilities in autonomous vehicles, as seen in models like ORION and VLM-RL [33][39].