Core Viewpoint - The article discusses the advancements in AI, particularly focusing on Huawei's innovations in the MoE (Mixture of Experts) architecture and the introduction of RL (Reinforcement Learning) post-training techniques, which aim to enhance the efficiency and performance of large language models (LLMs) in the competitive AI landscape [1][3]. Group 1: MoE Architecture and Huawei's Innovations - The MoE model, originally proposed by Canadian scholars, has evolved significantly, with Huawei introducing the MoGE architecture that addresses inefficiencies in the traditional MoE model, leading to cost reduction and improved training and deployment [1]. - Huawei's approach emphasizes the importance of creating a collaborative ecosystem to foster the growth of the Ascend ecosystem in China [1]. Group 2: RL Post-Training Techniques - RL post-training has emerged as a critical pathway to enhance LLM performance, with models like OpenAI's o1 and DeepSeek-R1 leveraging this technique to improve reasoning capabilities in complex tasks [3][5]. - The RL post-training phase currently consumes 20% of the total computational resources, projected to rise to 50%, significantly impacting model performance and costs [3]. Group 3: Challenges in RL Post-Training - The traditional On-Policy algorithms create a "computational black hole" due to the alternating execution of training and inference tasks, leading to underutilization of resources [6][7]. - The complexity of task scheduling in large-scale clusters, exacerbated by the adoption of various parallel strategies, poses significant challenges for efficient resource utilization [8]. Group 4: Innovations in Resource Utilization - Huawei's RL Fusion technology allows a single card to handle both training and inference tasks simultaneously, effectively doubling resource utilization and throughput [9][10]. - The StaleSync mechanism enables near-asynchronous execution of tasks, achieving over 90% efficiency in horizontal scaling across CloudMatrix 384 super nodes [16][20]. Group 5: Performance Metrics and Results - The combination of RL Fusion and StaleSync has led to a significant increase in efficiency, with single-node throughput improving by 78.5% and overall performance enhancement of 1.5 times [30][31]. - StaleSync's implementation in cluster scaling shows a linear throughput increase from 35k tokens/s to 127k tokens/s as the number of super nodes increases, demonstrating its effectiveness in enhancing scalability [32]. Group 6: Conclusion - The advancements in RL post-training techniques by Huawei represent a significant leap in AI efficiency, positioning the company as a key player in the next generation of AI technology [33].

Core Viewpoint - Reinforcement Learning (RL) post-training has become a crucial method for breaking through the performance ceiling of large language models (LLMs), with Huawei introducing two key technologies to enhance efficiency and resource utilization in this process [1][2][26]. Group 1: RL Post-Training Technologies - RL post-training now consumes 20% of the total computational power in the training process, projected to rise to 50%, significantly impacting model performance and costs [1]. - Huawei's "RL Fusion" technology allows a single card to handle both training and inference tasks simultaneously, doubling resource utilization and throughput [4][5]. - The "StaleSync" mechanism breaks the synchronization limitations, achieving over 90% efficiency in cluster expansion and a 50% increase in training throughput [2][10]. Group 2: Challenges in RL Post-Training - The traditional On-Policy algorithm requires alternating between training and inference tasks, leading to significant resource idling, especially in large-scale clusters [3]. - The complexity of task scheduling has increased exponentially with the popularity of Mixture of Experts (MoE) models, complicating resource utilization [4]. Group 3: Performance Improvements - RL Fusion technology enables dynamic switching between training and inference modes, optimizing memory usage and enhancing efficiency [5][8]. - The combination of RL Fusion and StaleSync technologies has led to a 78.5% increase in throughput for single supernodes, with overall performance improvements of 1.5 times [22][24]. - StaleSync allows for linear scalability in cluster expansion, with throughput increasing from 35k tokens/s to 127k tokens/s as the cluster size grows from one to four supernodes, achieving a linearity of 91% [24]. Group 4: Future Implications - The advancements in RL post-training technologies position Huawei's CloudMatrix 384 supernodes as a "super accelerator" for training large models, enhancing speed and efficiency significantly [2][26]. - The innovations in resource utilization and task parallelism are expected to drive the next generation of AI efficiency, marking a pivotal moment in the evolution of large model training [26].

Core Viewpoint - Reinforcement Learning (RL) post-training has become a crucial path for breaking through the performance ceiling of large language models (LLMs), with Huawei introducing two key technologies to enhance efficiency and resource utilization in this process [2][3][56]. Group 1: RL Post-Training Challenges - RL post-training currently consumes 20% of the total computational power in the training process, projected to rise to 50%, significantly impacting model performance and costs [3]. - Traditional RL post-training suffers from low resource utilization due to the alternating execution of training and inference tasks, leading to substantial computational waste [11][13]. - The complexity of task scheduling in large-scale clusters has increased due to the popularity of Mixture of Experts (MoE) models, making efficient collaboration challenging [15][16]. Group 2: Huawei's Innovations - Huawei's "RL Fusion" technology allows a single card to handle both training and inference tasks simultaneously, effectively doubling resource utilization and throughput [5][18]. - The "StaleSync" mechanism enables a quasi-asynchronous approach, allowing different RL tasks to execute in parallel within a defined "staleness threshold," improving horizontal scaling efficiency to over 90% [29][32]. - The combination of RL Fusion and StaleSync technologies significantly enhances the efficiency of RL post-training, achieving a throughput increase of 1.5 times [52][56]. Group 3: Performance Metrics - The implementation of RL Fusion can lead to a throughput increase from 14.0k tokens/sec to 35.0k tokens/sec when combined with StaleSync, representing a 150% improvement compared to baseline configurations [54]. - In a multi-node setup, StaleSync allows for linear scaling efficiency, with throughput increasing from 35k tokens/sec to 127k tokens/sec as the number of nodes increases from 1 to 4, achieving a linearity of 91% [55].