点击下方 卡片 ，关注" 具身智能 之心 "公众号 编辑丨具身智能之心，作者：张峻川 本文只做学术分享，如有侵权，联系删文 >> 点击进入→ 具身智能之心 技术交流群 更多干货，欢迎加入国内首个具身智能全栈学习社区 ： 具身智能之心知识星球 (戳我) ， 这里包含所有你想要的。 一、写在前面&背景 神经渲染（NERF/3DGS）引发了三维重建技术的革命，目前已经在辅助驾驶/具身智能领域得到大量应用。NERF和3DGS使用神经网络表达空间，其在新视角合成 方面的优越表现直击辅助驾驶/具身智能仿真的一大痛点：传感器仿真。如果这一类深度学习技术能够推广应用，就能够很大程度上解决传统计算机图形学渲染出的 图像缺乏真实性的问题，可以广泛应用在算法的闭环测试和训练中。 目前已经有一些研究项目在围绕NERF和3DGS技术打造全新的面向闭环测试的仿真框架。然而完全新开发一个仿真框架，使其具有现行场景仿真软件类似的功能将 会有巨大的工作量。因此另一个应用神经渲染新技术的思路是：将NERF和3DGS训练出的模型嵌入到现有仿真软件的框架中去，在保证实时渲染的前提下，同时能 够应用仿真软件已有的3D数字资产和算法接口等工具链。 在N ...

Core Viewpoint - 3D reconstruction is a critical intersection of computer vision and graphics, serving as the digital foundation for cutting-edge applications such as virtual reality, augmented reality, autonomous driving, and digital twins. Recent advancements in new perspective synthesis technologies, represented by Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have significantly improved reconstruction quality, speed, and dynamic adaptability [5][6]. Group 1: Introduction and Demand - The resurgence of interest in 3D reconstruction is driven by new application demands across various fields, including city-scale digital twins requiring kilometer-level coverage and centimeter-level accuracy, autonomous driving simulations needing dynamic traffic flow and real-time semantics, and AR/VR social applications demanding over 90 FPS and photo-realistic quality [6]. - Traditional reconstruction pipelines are inadequate for these new requirements, prompting the integration of geometry, texture, and lighting through differentiable rendering techniques [6]. Group 2: Traditional Multi-View Geometry Reconstruction - The traditional multi-view geometry approach (SfM to MVS) has inherent limitations in quality, efficiency, and adaptability to dynamic scenes, which have been addressed through iterative advancements in NeRF and 3DGS technologies [7]. - A comprehensive comparison of various methods highlights the evolution and future challenges in the field of 3D reconstruction [7]. Group 3: NeRF and Its Innovations - NeRF models scenes as continuous 5D functions, enabling advanced rendering techniques that have evolved significantly from 2020 to 2024, addressing issues such as data requirements, texture limitations, lighting sensitivity, and dynamic scene handling [13][15]. - Various methods have been developed to enhance quality and efficiency, including Mip-NeRF, NeRF-W, and InstantNGP, each contributing to improved rendering speeds and reduced memory usage [17][18]. Group 4: 3DGS and Its Advancements - 3DGS represents scenes as collections of 3D Gaussians, allowing for efficient rendering and high-quality output. Recent methods have focused on optimizing rendering quality and efficiency, achieving significant improvements in memory usage and frame rates [22][26]. - The comparison of 3DGS with other methods shows its superiority in rendering speed and dynamic scene reconstruction capabilities [31]. Group 5: Future Trends and Conclusion - The next five years are expected to see advancements in hybrid representations, real-time processing on mobile devices, generative reconstruction techniques, and multi-modal fusion for robust reconstruction [33]. - The ultimate goal is to enable real-time 3D reconstruction accessible to everyone, marking a shift towards ubiquitous computing [34].

Core Viewpoint - The article discusses the current state and future potential of end-to-end (E2E) autonomous driving systems, emphasizing the need for a shift from modular to E2E approaches in the industry, while acknowledging the challenges and limitations that still exist in achieving maturity in this technology [3][5]. Group 1: End-to-End Autonomous Driving - The concept of end-to-end systems involves directly processing raw sensor data to output control signals for vehicles, representing a significant shift from traditional modular approaches [3][4]. - E2E systems are seen as a way to provide a comprehensive representation of the information affecting vehicle behavior, which is crucial for handling the open-set scenarios of autonomous driving [4]. - The industry is currently divided, with some companies focusing on Vehicle Language Architecture (VLA) and others on traditional methods, but there is a consensus that E2E systems are the future [2][5]. Group 2: Industry Trends and Challenges - There is a growing recognition that autonomous driving is transitioning from rule-based to knowledge-driven systems, which necessitates a deeper understanding of E2E methodologies [5]. - Despite the high potential of E2E systems, there are still significant challenges to overcome before they can fully replace traditional planning and control methods [5]. - The article suggests that companies should allow more time for E2E systems to mature rather than rushing to implement them without adequate understanding [5]. Group 3: Community and Learning Resources - The "Autonomous Driving Heart Knowledge Planet" community aims to provide a platform for sharing knowledge and resources related to autonomous driving, including technical routes and job opportunities [8][18]. - The community has gathered over 4,000 members and aims to expand to nearly 10,000 within two years, offering a space for both beginners and advanced learners to engage with industry experts [8][18]. - Various learning resources, including video tutorials and technical discussions, are available to help members navigate the complexities of autonomous driving technologies [12][18].

Core Viewpoint - The article discusses the development and implementation of the Street Gaussians algorithm for dynamic scene representation in autonomous driving, highlighting its efficiency in training and rendering compared to previous methods [2][3]. Group 1: Background and Challenges - Previous methods faced challenges such as slow training and rendering speeds, as well as inaccuracies in vehicle pose tracking [3]. - Street Gaussians aims to generate realistic images for view synthesis in dynamic urban street scenes by modeling them as a combination of foreground moving vehicles and static backgrounds [3][4]. Group 2: Technical Implementation - The background model is represented as a set of points in world coordinates, each assigned a 3D Gaussian to represent geometry and color, with parameters optimized to avoid invalid values [8]. - The object model for moving vehicles includes a set of optimizable tracking poses and point clouds, with similar Gaussian attributes to the background model but defined in local coordinates [11]. - A 4D spherical harmonic model is introduced to encode temporal information into the appearance of moving vehicles without high storage costs [12]. Group 3: Initialization and Data Handling - Street Gaussians utilizes aggregated LiDAR point clouds for initialization, addressing the limitations of traditional SfM point clouds in urban environments [17]. - For objects with fewer than 2,000 LiDAR points, random sampling is employed to ensure sufficient data for model initialization [17]. Group 4: Course and Learning Opportunities - The article promotes a specialized course on 3D Gaussian Splatting (3DGS), covering various subfields and practical applications in autonomous driving, aimed at enhancing understanding and implementation skills [26][35].