在复杂真实场景中评估 π0 这类通用 policy 的性能和边界

Core Viewpoint - The article discusses the evaluation of the PI0-FAST-DROID model in real-world scenarios, highlighting its potential and limitations in robotic operations, particularly in handling new objects and tasks without extensive prior training [4][10][77]. Evaluation Method - The evaluation utilized the π₀-FAST-DROID model, specifically fine-tuned for the DROID robot platform, which includes a Franka Panda robot equipped with cameras [5][10]. - The assessment involved over 300 trials across various tasks, focusing on the model's ability to perform in diverse environments, particularly in a kitchen setting [10][11]. Findings - The model demonstrated a strong prior assumption of reasonable behavior, often producing intelligent actions, but these were not always sufficient to complete tasks [11]. - Prompt engineering was crucial, as variations in task descriptions significantly affected success rates, indicating the need for clear and structured prompts [12][59]. - The model exhibited impressive visual-language understanding and could mimic continuous actions across different scenarios [13][28]. Performance in Complex Scenarios - The model showed robust performance in recognizing and manipulating transparent objects, which is a significant challenge for traditional methods [20][27]. - It maintained focus on tasks despite human movement in the background, suggesting effective prioritization of relevant visual inputs [25]. Limitations - The model faced challenges with semantic ambiguity and often froze during tasks, particularly when it encountered unfamiliar commands or objects [39][42]. - It lacked memory, which hindered its ability to perform multi-step tasks effectively, leading to premature task completion or freezing [43][32]. - The model struggled with precise spatial reasoning, particularly in estimating distances and heights, which resulted in failures during object manipulation tasks [48][50]. Task-Specific Performance - The model's performance varied across different task categories, with notable success in simple tasks but significant challenges in complex operations like pouring liquids and interacting with household appliances [89][91][100]. - For instance, it achieved a 73.3% progress rate in pouring toy items but only 20% when dealing with real liquids, indicating limitations in physical capabilities [90]. Conclusion - The evaluation indicates that while the PI0 model shows promise as a generalist policy in robotic applications, it still requires significant improvements in instruction adherence, fine manipulation, and handling partial observability [77][88].