点击下方 卡片 ，关注" 具身智能 之心 "公众号 编辑丨具身智能之心 本文只做学术分享，如有侵权，联系删文 研究指出，灾难性遗忘的根源在于 VLM 的互联网级预训练数据与机器人微调数据之间存在 分布不匹配 —— 预训练数据以图文对为主，而机器人数据 以低维动作向量为主，这种差异迫使研究者采用全参数微调，进一步加剧了对预训练知识的覆盖。 论文：Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting 链接：https://arxiv.org/pdf/2509.22195 项目：https://vlm2vla.github.io/ VLM2VLA 在做什么？ VLM2VLA 的核心思路是 从数据层面解决分布不匹配问题 ，将低维动作转化为自然语言描述，使 VLA 微调数据与 VLM 预训练的图文分布对齐，进 而仅通过低秩适应（LoRA）微调即可实现动作生成，最小化对 VLM backbone 的修改，最终避免灾难性遗忘。 VLM2VLA训练范式首先通过自然语言表征底层动作，在数据层面解决分布失配问题。这种对齐机制 ...

点击下方 卡片 ，关注" 具身智能 之心 "公众号 作者丨 Dapeng Zhang等 编辑丨具身智能之心 本文只做学术分享，如有侵权，联系删文 >> 点击进入→ 具身智能之心 技术交流群 | | | 1. 介绍 机器人学长期以来一直是科学研究中的重要领域。早期的机器人主要依赖预编程的指令和人工设计的控制策略来完成任务分解与执行。这类方法通常应用于简 单、重复性的任务，例如工厂流水线和物流分拣。近年来，人工智能的快速发展使研究者能够在图像、文本和点云等多模态数据中，利用深度学习的特征提取与 轨迹预测能力。通过结合感知、检测、跟踪和定位等技术，研究者将机器人任务分解为多个阶段，以满足执行需求，从而推动了具身智能与自动驾驶的发展。然 而，大多数机器人仍然作为孤立的智能体存在，它们通常为特定任务而设计，缺乏与人类和外部环境的有效交互。 为克服这些局限性，研究者开始探索将大语言模型（LLMs）与视觉语言模型（VLMs）引入机器人操作中，以实现更精准和灵活的控制。现代的机器人操作方法 通常依赖视觉-语言生成范式（如自回归模型 或扩散模型），并结合大规模数据集 以及先进的微调策略。我们将这些方法称为 VLA基础模型，它们 ...

端到端学习已迅速成为自动驾驶的基础范式，促进了感知、预测和规划在统一框架下的联合优化。借助大规模驾驶数据集，这些模型能够直接从原始传感器数据中学习 驾驶策略，在各种真实场景中展现出令人印象深刻的性能。然而，尽管取得了这些进展，当前的E2E方法仍面临持续的挑战：它们往往难以泛化到稀有的长尾事件，对 高级场景语义理解不足，并且缺乏在开放世界环境中所需的自适应且可解释的推理能力。 与此同时，大型语言模型（LLMs）和视觉语言模型（VLMs）的出现，凸显了它们在上下文学习、常识推理和超越训练分布的泛化能力方面的卓越表现。这些新兴能力 为提升自动驾驶系统的智能性和鲁棒性提供了极具吸引力的机会，特别是在面对真实世界、安全关键的部署复杂性时。然而，直接将现有的VLM应用于自动驾驶存在显 著挑战。大多数VLM主要针对静态二维视觉语言任务进行优化，限制了其在丰富、动态的三维驾驶环境中的空间推理和全面场景理解能力。更关键的是，缺乏显式的时 间建模机制使得这些模型无法有效推理随时间展开的交互、运动和因果关系。此外，它们倾向于产生幻觉式或不可靠的描述，严重影响了自动驾驶等高风险应用所需的 可信度。因此，一个重要的技术难题浮现出来：如 ...

Core Viewpoint - The article discusses the advancements in multimodal fusion and vision-language models (VLMs) in robot vision, emphasizing their role in enhancing robots' perception and understanding capabilities in complex environments [4][5][56]. Multimodal Fusion in Robot Vision Tasks - Semantic scene understanding is a critical task in visual systems, where multimodal fusion significantly improves accuracy and robustness by integrating additional information such as depth and language [9][11]. - Current mainstream fusion strategies include early fusion, mid-level fusion, and late fusion, evolving from simple concatenation to more sophisticated interactions within a unified architecture [10][12][16]. Applications of Multimodal Fusion - In autonomous driving, 3D object detection is crucial for accurately identifying and locating pedestrians, vehicles, and obstacles, with multimodal fusion enhancing environmental understanding [15][18]. - The design of multimodal fusion involves addressing when to fuse, what to fuse, and how to fuse, with various strategies impacting performance and computational efficiency [16][17]. Embodied Navigation - Embodied navigation allows robots to explore and act in real environments, focusing on autonomous decision-making and dynamic adaptation [23][25][26]. - Three representative methods include goal-directed navigation, instruction-following navigation, and dialogue-based navigation, showcasing the evolution from perception-driven to interactive understanding [25][26][27]. Visual Localization and SLAM - Visual localization determines a robot's position, which is challenging in dynamic environments; recent methods leverage multimodal fusion to improve performance [28][30]. - SLAM (Simultaneous Localization and Mapping) has evolved from geometric-driven to semantic-driven approaches, integrating various sensor data for enhanced adaptability [30][34]. Vision-Language Models (VLMs) - VLMs have progressed significantly, focusing on semantic understanding, 3D object detection, embodied navigation, and robot operation, with various fusion methods being explored [56][57]. - Key innovations in VLMs include large-scale pre-training, instruction fine-tuning, and structural optimization, enhancing their capabilities in cross-modal reasoning and task execution [52][53][54]. Future Directions - Future research should focus on structured spatial modeling, improving system interpretability and ethical adaptability, and developing cognitive VLM architectures for long-term learning [57][58].

Core Viewpoint - The article introduces the Kitchen-R benchmark, a unified evaluation framework for task planning and low-level control in embodied AI, addressing the existing fragmentation in current benchmarks [4][6][8]. Group 1: Importance of Benchmarks - Benchmarks are crucial in various fields such as natural language processing and computer vision for assessing model progress [7]. - In robotics, simulator-based benchmarks like Behavior-1K are common, providing model evaluation and training capabilities [7]. Group 2: Issues with Existing Benchmarks - Current benchmarks for high-level language instruction and low-level robot control are fragmented, leading to incomplete assessments of integrated systems [8][9]. - High-level benchmarks often assume perfect execution of atomic tasks, while low-level benchmarks rely on simple single-step instructions [9]. Group 3: Kitchen-R Benchmark Features - Kitchen-R fills a critical gap in embodied AI research by providing a comprehensive testing platform that closely simulates real-world scenarios [6][8]. - It includes a digital twin kitchen environment and over 500 language instructions, supporting mobile ALOHA robots [9][10]. - The benchmark supports three evaluation modes: independent evaluation of planning modules, independent evaluation of control strategies, and critical full system integration evaluation [9][10]. Group 4: Evaluation Metrics - Kitchen-R is designed with offline independent evaluation and online joint evaluation metrics to ensure comprehensive system performance measurement [16][20]. - Key metrics include Exact Match (EM) for task planning accuracy and Mean Squared Error (MSE) for trajectory prediction accuracy [20][21]. Group 5: Baseline Methods - Kitchen-R provides two baseline methods: a VLM-driven task planning baseline and a Diffusion Policy low-level control baseline [43][49]. - The VLM planning baseline enhances planning accuracy through contextual examples and constrained generation [47][48]. - The Diffusion Policy baseline integrates visual features and robot states to predict future actions [49][52]. Group 6: Future Directions - Kitchen-R can expand to include more complex scenarios, such as multi-robot collaboration and dynamic environments, promoting the application of language-guided mobile manipulation robots in real-world settings [54].

Core Insights - The article discusses the advancements in multimodal fusion and vision-language models (VLMs) as essential tools for enhancing robot vision technology, emphasizing their potential in complex reasoning and long-term task decision-making [4][10]. Multimodal Fusion and Robot Vision - Multimodal fusion enhances semantic scene understanding by integrating various data sources, such as visual, linguistic, depth, and lidar information, addressing limitations faced by traditional unimodal methods [8][9]. - The rise of VLMs has propelled the development of multimodal fusion paradigms, showcasing capabilities in zero-shot understanding and instruction following [9][10]. Key Applications and Challenges - The article identifies key applications of multimodal fusion in tasks like simultaneous localization and mapping (SLAM), 3D object detection, navigation, and robot manipulation [10][19]. - Challenges in multimodal fusion include cross-modal alignment, efficient training strategies, and real-time performance optimization [10][19]. Data Sets and Benchmarking - A comprehensive analysis of mainstream multimodal datasets used for robot tasks is provided, detailing their modality combinations, task coverage, and limitations [10][43]. - The importance of high-quality multimodal datasets is highlighted, as they are crucial for model training and performance evaluation [62]. Future Directions - The article suggests future research directions to address challenges in multimodal fusion, such as improving cross-modal alignment techniques and enhancing real-time performance [10][63]. - Emphasis is placed on the need for standardized datasets and benchmarks to facilitate comparisons across different research efforts [66].

Core Insights - The article discusses the potential of large language models (LLMs) in robotic program planning, highlighting their ability to generate coherent action sequences but also noting their limitations in providing the necessary sensory details for physical execution [3][4] - It introduces a new framework called SelfReVision, which enhances the performance of small visual language models (VLMs) through self-distillation without external supervision, aiming to improve their planning capabilities in real-world scenarios [4][9] Research Background - LLMs show promise in generating action sequences but often lack the precision required for robotic tasks due to their reliance on human-centric training data [3] - Visual language models (VLMs) can potentially address these limitations, but existing methods either require specialized simulation environments or are costly to train and deploy [3] Methodology - SelfReVision is proposed as a self-improvement framework that allows small VLMs to enhance their performance through iterative self-critique and revision [4][6] - The framework operates in three stages: critique, revise, and verify, enabling models to generate and refine plans based on self-assessment [4][10] Experimental Setup - Two types of experiments were conducted to evaluate the planning capabilities of SelfReVision: image-based program planning and entity-agent tasks [11] - Evaluation metrics included coverage, ordering, completeness, overall quality, and a new metric called image groundedness [12] Key Results - SelfReVision significantly outperformed baseline models across various metrics, achieving an average win rate of 68% on the PLACES dataset and 72% on the SIMULATION dataset [13] - Larger models benefited more from SelfReVision, with an average gain of 74% for models with 12 billion parameters or more [13] Comparison with Other Methods - SelfReVision demonstrated clear advantages over other methods like Best-of-N and PaliGemma, with improvements of 60% in most settings compared to modest gains from Best-of-N [17] - When compared to GPT-4o, SelfReVision's plans had at least a 25% higher win rate for models with 12 billion parameters or more, indicating its effectiveness in enhancing smaller models [17] Ablation Studies - The complete Criticize-Revise-Verify (CRV) process showed the strongest performance, with average win rates of 68.3% on the PLACES dataset and 71.9% on the SIMULATION dataset [18] - Variants of the process showed significant performance drops, emphasizing the importance of the verification step in filtering out suboptimal revisions [18] Application in Entity-Agent Tasks - SelfReVision was tested in challenging scenarios, showing a 26% improvement for the Gemma 12B model and a 17% improvement for the Gemma 27B model in block manipulation tasks [21] - In hierarchical tasks, SelfReVision plans led to a 70% success rate in generating trajectories, surpassing the 61% success rate of baseline models [21]

Core Insights - The article discusses the limitations of current Vision-Language Models (VLMs) in spatial reasoning tasks, highlighting the need for improved datasets and methodologies to enhance performance in various scenarios [3][12]. Dataset Limitations - The existing InternSpatial dataset has three main limitations: 1. Limited scene diversity, focusing primarily on indoor and outdoor environments, lacking diverse contexts like driving and embodied navigation [3]. 2. Restricted instruction formats, only supporting natural language or region masks, which do not encompass the variety of queries found in real-world applications [3]. 3. Lack of multi-view supervision, with over 90% of data focusing on single-image reasoning, failing to model spatiotemporal relationships across views [3]. Evaluation Benchmark - The InternSpatial-Bench evaluation benchmark includes 6,008 QA pairs across five tasks, assessing position comparison, size comparison, rotation estimation, object counting, and existence estimation [7]. - The benchmark also introduces 1,000 additional QA pairs for multi-view rotation angle prediction [7]. Data Engine Design - The data engine employs a three-stage automated pipeline: 1. Annotation generation using existing annotations or SAM2 for mask generation [9]. 2. View alignment to construct a standard 3D coordinate system [9]. 3. Template-based QA generation with predefined task templates [9]. Experimental Results - Spatial reasoning performance has improved, with InternVL-Spatial-8B showing a 1.8% increase in position comparison accuracy and a 17% increase in object counting accuracy compared to its predecessor [10]. - The model's performance across various tasks demonstrates significant enhancements, particularly in multi-view tasks [10]. Instruction Format Robustness - Current models exhibit a 23% accuracy drop when using the <box> format, while training with InternSpatial reduces the gap between different formats to within 5% [12]. - However, the automated QA generation struggles to replicate the complexity of natural language, indicating a need for further refinement [12].

Group 1 - The core issue in embodied intelligence is the lack of long-term memory, which limits the ability to process multimodal observational data across time and space [3] - Current visual language models (VLMs) excel in planning and control tasks but struggle with integrating historical experiences in embodied environments [3][5] - Existing video QA benchmarks fail to adequately assess tasks requiring fine-grained reasoning, such as object manipulation and navigation [5] Group 2 - The proposed benchmark includes a task architecture that allows for dynamic environment interaction and memory reasoning validation [4][6] - A total of 60 task categories are designed to cover spatiotemporal semantic memory challenges, including spatial relations, temporal reasoning, attribute memory, and multi-target recall [7] - Key technical innovations include a programmatic expansion of task complexity through increased interaction counts and a strict separation of experience collection from interaction phases [9][6] Group 3 - Experimental results reveal three major bottlenecks in VLM memory capabilities across 60 tasks, including failures in long-sequence reasoning, weak spatial representation, and collapse in multi-target processing [13][14][16] - The performance of native VLMs declines as the number of frames increases, indicating ineffective utilization of long contexts [20] - Supervised fine-tuning models show improved performance by leveraging longer historical data, suggesting a direction for VLM refinement [25] Group 4 - The benchmark represents the first photorealistic embodied memory evaluation framework, covering complex household environments and allowing for scalable assessment [26] - Future directions include memory compression techniques, end-to-end joint training to address the split between high-level reasoning and low-level execution, and the development of long-term video understanding [26]