Workflow
多模态场景理解
icon
Search documents
CVPR 2025 | 多模态统一学习新范式来了,数据、模型、代码全部开源
机器之心· 2025-06-12 00:53
Core Viewpoint - The article discusses the development of a unified audio-visual scene understanding model, emphasizing the need for models to possess general understanding capabilities similar to humans, rather than focusing solely on single-task performance [2][13]. Group 1: Unified Learning Paradigm - The current mainstream learning paradigm for multi-modal models overlooks the heterogeneity of multi-modal data and the complex relationships between tasks, leading to potential interference when tasks are jointly trained [2][13]. - A new paradigm for multi-modal large model learning is proposed, focusing on effective task cooperation from both data and model perspectives, surpassing specialized models in various scene understanding tasks [3][14]. Group 2: Data and Model Structure - The AV-UIE dataset is introduced, which includes explicit reasoning processes and specific temporal and spatial information to clarify the mutual assistance relationships between tasks [15][16]. - The proposed model architecture includes interaction-aware LoRA structures that allow for the decoupling of different capabilities, enabling tasks to share and benefit from enhanced abilities [21][23]. Group 3: Experimental Results - The Crab model demonstrates superior general understanding capabilities across multiple tasks compared to other models, as evidenced by comprehensive ablation studies and performance comparisons [26][30]. - The model outperforms specialized models in tasks such as AVE, ARIG, and AVQA, showcasing its effectiveness in enhancing performance through task cooperation [27][28][29].