Core Viewpoint - The article discusses the challenges of long text modeling in large language models (LLMs) and introduces a new attention mechanism called Grouped Cross Attention (GCA) that enhances the ability to process long contexts efficiently, potentially paving the way for advancements in artificial general intelligence (AGI) [1][2]. Long Text Processing Challenges and Existing Solutions - Long text modeling remains challenging due to the quadratic complexity of the Transformer architecture and the limited extrapolation capabilities of full-attention mechanisms [1][6]. - Existing solutions, such as sliding window attention, sacrifice long-range information retrieval for continuous generation, while other methods have limited generalization capabilities [7][8]. GCA Mechanism - GCA is a novel attention mechanism that learns to retrieve and select relevant past segments of text, significantly reducing memory overhead during long text processing [2][9]. - The mechanism operates in two stages: first, it performs attention on each chunk separately, and then it fuses the information from these chunks to predict the next token [14][15]. Experimental Results - Models incorporating GCA demonstrated superior performance on long text datasets, achieving over 1000 times length generalization and 100% accuracy in 16M long context retrieval tasks [5][17]. - The GCA model's training costs scale linearly with sequence length, and its inference memory overhead approaches a constant, maintaining efficient processing speeds [20][21]. Conclusion - The introduction of GCA represents a significant advancement in the field of long-context language modeling, with the potential to facilitate the development of intelligent agents with permanent memory capabilities [23].

琶洲实验室、华南理工大学联合推出关键上下文感知注意力机制（CCA-Attention），实现超长文本的高效上下文建模。在 128K 超长序列上下文建模任 务中，CCA-Attention 的推理速度是标准自注意力机制的 7.9 倍，同时键值缓存（KV Cache）显存占用减少 93%，性能全面优于现有高效注意力方法。 该成果已被 ICML 2025 接收，最早于 2024 年 12 月 17 日提交至 ArXiv，早于 DeepSeek NSA 和 Kimi MoBA 公开。CCA-Attention 不仅速度快、 资源占用低，更在上下文建模的精准度和效率上树立了新标杆，为长文本处理注入全新动力。 引言 近期研究 [1, 2, 3] 发现，LLMs 中的大多数层的注意力权重主要集中在少数 token 上，表现出显著的稀疏性（见图 1）。这一发现启示我们可以借助这种 稀疏特性，降低注意力机制的计算复杂度。 图 1： LLaMA2-7B 模型中注意力权重的可视化，阴影越深表示注意力权重越高。最后一个 token 仅对上下文少数几个 token 有着较高的注意力权重，即注意力权重具有 显著的稀疏性 。 现有稀疏注意 ...