Core Insights - The article discusses the advancements and challenges in large language models (LLMs), emphasizing their transformative impact on human-computer interaction and the need for efficient architectures to overcome high training and inference costs [2][3][8]. Group 1: LLM Architecture and Efficiency - The efficiency of LLMs is primarily attributed to the Transformer architecture, which, despite its breakthroughs, faces challenges due to its O(N^2) complexity in long sequence tasks [3][4]. - Recent innovations in Transformer architecture have emerged, but a comprehensive review summarizing these advancements has been lacking [4][5]. - A collaborative effort by Shanghai AI Lab and several institutions has resulted in a survey of over 440 papers, focusing on the latest progress in efficient LLM architectures [5][6]. Group 2: Categories of Efficient Architectures - The survey categorizes efficient LLM architectures into seven types, including linear sequence modeling, sparse sequence modeling, efficient full attention, sparse expert models, mixed model architectures, diffusion language models, and applications to other modalities [6][8]. - Linear sequence modeling aims to reduce attention training and inference complexity without incurring KV cache overhead [6][8]. - Sparse sequence modeling leverages the inherent sparsity of attention maps to accelerate computation [21][22]. Group 3: Innovations in Attention Mechanisms - Efficient full attention methods optimize memory access and KV storage while maintaining complete attention [22][23]. - Sparse expert models enhance model capacity without proportionally increasing computational costs through conditional activation of experts [27][28]. - Mixed architectures find a balance between linear/sparse attention and full attention, optimizing both efficiency and performance [35][36]. Group 4: Applications and Future Directions - Diffusion language models represent a novel approach by applying diffusion models from visual tasks to language generation, significantly improving generation speed [38][39]. - Efficient architectures are being applied across various modalities, including vision and audio, demonstrating their versatility and effectiveness [44][45]. - The overarching goal is to achieve substantial acceleration in AI development, akin to the phrase "Speed Always Wins," suggesting a focus on efficiency in training and deploying powerful models [45].

Core Insights - The article highlights the rise of Linear-MoE architecture, which effectively combines linear sequence modeling and Mixture-of-Experts (MoE) for enhanced performance in large language models [1][10]. Group 1: Linear Sequence Modeling - Significant advancements in linear sequence modeling have been achieved over the past two years, characterized by linear time complexity in training and constant memory usage during inference [5]. - The main categories of linear sequence modeling include Linear Attention, State Space Models (SSM), and Linear RNN, with notable works such as Lightning Attention, GLA, Mamba2, and RWKV [5]. Group 2: Mixture-of-Experts (MoE) - MoE has become a standard in the industry, with various models like GPT-4, Gemini, and domestic models such as DeepSeek and Qwen all adopting MoE architectures [8]. - The importance of MoE in enhancing model capabilities is emphasized, although the article does not delve deeply into this aspect [8]. Group 3: Linear-MoE Architecture - Linear-MoE offers a complete system from modeling to training, allowing flexible combinations of linear sequence modeling layers and MoE layers, while also being compatible with traditional Softmax Attention Transformer layers [10]. - Key features include a modular architecture with support for various linear modeling methods and multiple MoE implementations, ensuring stability and scalability through the Megatron-Core framework [10]. Group 4: Performance and Future Prospects - Large-scale experiments validate the superiority of Linear-MoE, demonstrating faster inference speeds (2-5 times quicker than traditional architectures) and over 50% reduction in memory usage [12][13]. - The open-source nature of Linear-MoE fills a technical gap and provides reproducible training solutions, with future exploration planned for applications in long-context understanding and Vision-Language model architectures [13].