Core Viewpoint - The rapid development of large models has made the Mixture of Experts (MoE) model a significant direction for expanding model capabilities due to its unique architectural advantages. However, training efficiency in distributed cluster environments remains a critical challenge that needs to be addressed [1][2]. Group 1: MoE Model Challenges - The training efficiency of MoE models faces two main challenges: (1) Expert parallelism introduces computational and communication waiting times, especially when the model size is large, leading to idle computational units waiting for communication [2][3]. (2) Load imbalance results in some experts being frequently called while others remain underutilized, causing further waiting among computational units [2]. Group 2: Optimization Solutions - Huawei has developed an optimization solution called Adaptive Pipe & EDPB, which aims to eliminate waiting times in MoE training systems by improving communication and load balancing [3][10]. - The AutoDeploy simulation platform allows for rapid analysis of diverse training loads and automatically identifies optimal strategies that match cluster hardware specifications, achieving a 90% accuracy rate in training performance [4]. Group 3: Communication and Load Balancing Innovations - The Adaptive Pipe communication framework achieves over 98% communication masking, allowing computations to proceed without waiting for communication [6][7]. - EDPB global load balancing enhances training efficiency by 25.5% by ensuring balanced expert scheduling during the training process [10]. Group 4: Dynamic Load Balancing Techniques - The team introduced expert dynamic migration technology, which allows for intelligent movement of experts between distributed devices based on predicted load trends, thus addressing load imbalance issues [12][14]. - A dynamic data rearrangement scheme was proposed to minimize computation time without sacrificing training accuracy, achieving load balancing during pre-training [14]. Group 5: Overall System Benefits - The combination of Adaptive Pipe & EDPB has led to a 72.6% increase in end-to-end training throughput for the Pangu Ultra MoE 718B model, demonstrating significant improvements in training efficiency [17].