港理&清华等首个具身程序性综述：让机器人从第一人称视角学习步骤、纠错与问答

Core Viewpoint - The article presents a comprehensive overview of the concept of an Egocentric Procedural AI Assistant (EgoProceAssist), which aims to assist individuals in performing daily procedural tasks from a first-person perspective. It identifies three core technical tasks necessary for this assistant: Egocentric Procedural Error Detection, Egocentric Procedural Learning, and Egocentric Procedural Question Answering [6][32]. Summary by Sections Motivation - The article emphasizes the prevalence of procedural tasks in daily life, which require a specific sequence of steps to achieve desired outcomes. It highlights the potential of an AI assistant to enhance safety and efficiency in performing these tasks, especially in high-risk scenarios [6][8]. New Classification System - A novel classification system is introduced, categorizing the three core tasks of the AI assistant and summarizing existing methods, datasets, and evaluation metrics relevant to each task [2][6]. Egocentric Procedural Error Detection - This section outlines the existing key technologies for detecting procedural errors from a first-person perspective. It differentiates between methods that require only video data and those that utilize multimodal data, emphasizing the unique challenges of procedural error detection compared to general anomaly detection [9][11][12]. Egocentric Procedural Learning - The article discusses various approaches to procedural learning, categorized by supervision levels: unsupervised, weakly supervised, and self-supervised methods. It highlights the importance of identifying key steps in procedural tasks to improve error detection and planning capabilities [14][16]. Egocentric Procedural Question Answering - This section summarizes current technologies for answering procedural questions from a first-person perspective, noting the challenges posed by occlusions and scene changes. It emphasizes the need for models to possess strong understanding and memory capabilities to effectively respond to user queries [17][20]. Supplementary Experiments - The article presents supplementary experiments that evaluate the performance of existing VLMs and AI agents in procedural error detection and learning tasks. The results indicate significant limitations in their ability to assist with first-person procedural tasks [23][25]. Challenges - The article identifies several challenges in developing the EgoProceAssist, including data scarcity, limited understanding of long-term procedural activities, and heavy reliance on manual annotations, which hinder real-time assistance capabilities [29][30][31]. Conclusion - The research concludes by reiterating the significance of the proposed AI assistant and its core tasks, while also addressing the ongoing challenges and limitations in the field. It aims to provide a foundation for future research directions in egocentric AI applications [32].