打破资源瓶颈！华南理工&北航等推出SEA框架：低资源下实现超强多模态安全对齐

Core Viewpoint - The article discusses the SEA framework (Synthetic Embedding for Enhanced Safety Alignment) developed by the team at Beihang University, which addresses the low-resource safety alignment challenges of multimodal large language models (MLLMs) by using synthetic embeddings instead of real multimodal data [1][2][3]. Summary by Sections Introduction - The SEA framework innovatively replaces real multimodal data with synthetic embeddings, providing a lightweight solution for the safe deployment of large models [1]. Challenges in MLLM Safety Alignment - MLLMs face three main challenges in safety alignment: 1. Reducing the cost of constructing multimodal safety alignment datasets [4]. 2. Overcoming the limitations of text alignment methods in non-text modal attack scenarios [5]. 3. Providing a universal safety alignment solution for emerging modalities [6]. SEA Framework Overview - SEA synthesizes embeddings from the representation space of modal encoders, allowing for cross-modal safety alignment using only text input, thus overcoming the high costs and strong modality dependencies of real data [6][8]. Data Preparation - The framework requires a text safety alignment dataset containing harmful instructions, which are used to optimize a set of embedding vectors [12]. Embedding Optimization - The optimization process aims to maximize the probability of the MLLM generating specified outputs based on the optimized embeddings, while keeping the MLLM parameters frozen [16][17]. Safety Alignment Implementation - To integrate the embedding vectors with the text dataset, specific prefixes are added to the text instructions, allowing for the construction of multimodal datasets for safety alignment training [19]. VA-SafetyBench: Safety Evaluation Benchmark - VA-SafetyBench is a safety evaluation benchmark for MLLMs that includes video and audio safety assessments, expanding upon existing image safety benchmarks [20][21]. Experimental Results - The SEA framework demonstrated effectiveness in reducing the success rate of multimodal attacks compared to traditional methods, particularly in complex attack scenarios involving images, videos, and audio [32][36]. Conclusion - The SEA framework shows promise as a solution for the safety alignment of emerging MLLMs, allowing for effective multimodal safety alignment using synthetic embeddings, which significantly reduces resource requirements [37].