DeepSeek连发两篇论文背后，原来藏着一场学术接力

Core Insights - The article discusses the evolution of DeepSeek's research, particularly focusing on their recent papers, mHC and Conditional Memory, which build upon previous works by ByteSeed and others in the field of AI and deep learning [1][2]. Group 1: mHC and Its Innovations - mHC builds on the Hyper-Connections (HC) framework proposed by ByteSeed, significantly improving the stability and scalability of deep learning models [4][7]. - The core innovation of mHC lies in its ability to expand the width of residual flows and introduce dynamic hyper connections, which enhances model capacity without increasing computational costs [4][6]. - mHC addresses the stability issues encountered in HC during large-scale training by implementing manifold constraints and optimizing infrastructure, making it suitable for industrial applications with trillions of parameters [7][8]. Group 2: Conditional Memory and N-gram Utilization - The Conditional Memory paper introduces the concept of using an "Engram" to allow models to reference a large phrase dictionary, improving efficiency in answering straightforward questions [9][12]. - This approach contrasts with previous methods by suggesting that integrating N-gram lookups can free up computational resources for more complex reasoning tasks [10][13]. - DeepSeek's research indicates that allocating a portion of parameters to Engram can yield better performance than solely relying on Mixture of Experts (MoE) models, revealing a U-shaped scaling law [13][19]. Group 3: Collaborative Research and Community Impact - The collaboration between DeepSeek and ByteSeed exemplifies the value of open research in advancing AI technologies, showcasing how shared insights can lead to significant breakthroughs [19][20]. - The article highlights various innovative approaches from ByteSeed, such as UltraMem and Seed Diffusion Preview, which contribute to the ongoing evolution of deep learning architectures [20].