从 DeepSeek 部署看，华为如何让 MOE 架构“迎来”海量“专家”？

Core Viewpoint - The development of models has shifted from early algorithm optimization to deep innovation at the system engineering level, transitioning from a digital era of bit traffic to a Token economy, with daily Token consumption in China rising from hundreds of billions to tens of trillions [1] Group 1: Model Optimization - Huawei has made significant optimizations for DeepSeek, focusing on three main areas to enhance compatibility and support for enterprise applications [3] - The pre-training aspect includes the implementation of DualPipe technology, which has been improved to minimize static memory usage through the introduction of the DualPipe-V solution [6] - At the operator level, Huawei has enhanced execution efficiency with the MRN PO fusion operator and optimized low-latency communication [7] Group 2: System Architecture - Huawei has developed a new architecture for inference called the "super node" architecture, which interconnects multiple GPUs to reduce communication latency and improve training throughput [14] - The Atlas 900 A3 SuperCluster has been designed to enhance cluster computing efficiency and reliability, achieving a training efficiency increase of 2.7 times [15] - The OmniPlacement algorithm has been introduced to optimize resource utilization by dynamically adapting to expert activation data, improving throughput by 10% [19] Group 3: Load Balancing and Efficiency - Huawei has implemented a large-scale expert parallel (large EP) strategy to enhance inference efficiency, achieving a nearly 20-fold increase in the past two months [17] - The company has developed dynamic priority adjustment and communication optimization strategies to address load balancing challenges in expert parallelism [20]