NSA

Search documents
刚刚,DeepSeek梁文锋NSA论文、北大杨耀东团队摘得ACL 2025最佳论文
机器之心· 2025-07-30 16:25
Group 1 - The ACL conference is a premier event in the field of computational linguistics and natural language processing, with the 63rd edition scheduled for July 27 to August 1, 2025, in Vienna, Austria [2] - This year, the total number of submissions reached a record high of over 8,000, compared to 4,407 last year, with acceptance rates of 20.3% for main conference papers and 16.7% for Findings [3] - Over half of the first authors of the submitted papers are from China (51.3%), a significant increase from last year's 30.6%, while the second-largest group of authors comes from the United States at 14.0% [4] Group 2 - Four best papers were awarded, including two from teams led by Liang Wenfeng and Yang Yaodong from Peking University, with the other two awarded to teams from CISPA Helmholtz Center for Information Security & TCS Research & Microsoft, and Stanford University & Cornell Tech [6][10] - The first best paper discusses a theory of response sampling in large language models (LLMs), highlighting the ethical concerns arising from biases in decision-making processes influenced by LLMs [11][15] - The second best paper focuses on algorithmic fairness, introducing a framework that emphasizes group discrimination awareness in specific contexts, demonstrating that existing bias mitigation strategies may be counterproductive [16][19] Group 3 - The third best paper reveals a structural inertia mechanism in large models that resists alignment during fine-tuning, indicating that achieving robust alignment is more challenging than previously thought [24][25] - The fourth best paper presents a new hardware-aligned and natively trainable sparse attention mechanism, which significantly improves efficiency in long-context modeling for LLMs [31][40] Group 4 - A total of 26 outstanding papers were recognized, covering various topics such as multilingual summarization, hate speech analysis, and the evaluation of large language models [42] - The best demo paper was awarded to OLMoTrace, a system capable of tracing language model outputs back to trillions of training tokens [46][48] Group 5 - The ACL 2025 conference also recognized two time-tested awards, celebrating foundational papers from 2000 and 2015 that have significantly influenced the field [65][73] - Kathy McKeown received the Lifetime Achievement Award for her extensive contributions to natural language processing over 43 years [86][90] - Julia B. Hirschberg was awarded the Distinguished Service Award for her long-standing service to the ACL and contributions to the field [96][98]
ICML 2025 | 千倍长度泛化!蚂蚁新注意力机制GCA实现16M长上下文精准理解
机器之心· 2025-06-13 15:45
Core Viewpoint - The article discusses the challenges of long text modeling in large language models (LLMs) and introduces a new attention mechanism called Grouped Cross Attention (GCA) that enhances the ability to process long contexts efficiently, potentially paving the way for advancements in artificial general intelligence (AGI) [1][2]. Long Text Processing Challenges and Existing Solutions - Long text modeling remains challenging due to the quadratic complexity of the Transformer architecture and the limited extrapolation capabilities of full-attention mechanisms [1][6]. - Existing solutions, such as sliding window attention, sacrifice long-range information retrieval for continuous generation, while other methods have limited generalization capabilities [7][8]. GCA Mechanism - GCA is a novel attention mechanism that learns to retrieve and select relevant past segments of text, significantly reducing memory overhead during long text processing [2][9]. - The mechanism operates in two stages: first, it performs attention on each chunk separately, and then it fuses the information from these chunks to predict the next token [14][15]. Experimental Results - Models incorporating GCA demonstrated superior performance on long text datasets, achieving over 1000 times length generalization and 100% accuracy in 16M long context retrieval tasks [5][17]. - The GCA model's training costs scale linearly with sequence length, and its inference memory overhead approaches a constant, maintaining efficient processing speeds [20][21]. Conclusion - The introduction of GCA represents a significant advancement in the field of long-context language modeling, with the potential to facilitate the development of intelligent agents with permanent memory capabilities [23].
大模型 “注意力简史”:与两位 AI 研究者从 DeepSeek、Kimi 最新改进聊起
晚点LatePost· 2025-03-02 06:10
嘉宾 丨 肖朝军、傅天予 整理 丨 程曼祺 上周,DeepSeek、Kimi 都放出了新的大模型架构改进和优化成果,分别是 NSA、MoBA。二者都聚焦对大 模型中 "注意力机制" 的改进。 o 1 、 R 1 等 推 理 模 型 的 出 现,给 了 长 文 本 新 课 题 。 注意力机制是当前大语言模型(LLM)的核心机制。2017 年 6 月那篇开启大语言模型革命的 Transformer 八 子论文,标题就是:Attention Is All You Need(注意力就是你所需要的一切)。 而优化 Attention 的计算效率和效果,又能帮助解决 AI 学界和业界都非常关心的一个问题,就是长文本(long context)。 不管是要一次输入一整本书,让模型能帮我们提炼、理解;还是在生成现在 o1、R1 这类模型需要的长思维 链;又或者是希望模型未来能有越来越长的 "记忆",这都需要长文本能力的支持。 这期节目我们邀请了两位做过 Attention 机制改进的 AI 研究者做嘉宾。 一位是清华计算机系自然语言处理实验室的博士生肖朝军,他是 InfLLM 注意力机制改进的一作,导师是清华 计算机系副教授 ...
月之暗面 MoBA 核心作者自述:一个 “新晋大模型训练师” 的三入思过崖
晚点LatePost· 2025-02-20 14:21
"从开源论文、开源代码出发,现在已经进化到开源思维链了嘛!" 文丨Andrew Lu 注释丨贺乾明 程曼祺 2 月 18 日,Kimi 和 DeepSeek 同一天发布新进展,分别是 MoBA 和 NSA,二者都是对 "注意力机 制"(Attention Mechanism)的改进。 今天,MoBA 的一位主要研发同学 Andrew Lu 在知乎发帖,自述研发过程的三次踩坑,他称为 "三入思过 崖"。他在知乎的签名是"新晋 LLM 训练师"。 这条回答下的一个评论是:"从开源论文、开源代码出发,现在已经进化到开源思维链了嘛。" 注意力机制之所以重要,是因为它是当前大语言模型(LLM)的核心机制。回到 2017 年 6 月那篇开启 LLM 革命的 Transformer 八子论文,标题就是:Attention Is All You Need(注意力就是你所需要的一 切),该论文被引用次数至今已达 15.3 万。 注意力机制能让 AI 模型像人类一样,知道在处理信息时该 "重点关注" 什么、"忽略" 什么,抓住信息中最 关键的部分。 在大模型的训练阶段和使用(推理)阶段,注意力机制都会发挥作用。它的大致工作原理是 ...