Core Viewpoint - The article discusses the advancements in AI computing clusters, particularly focusing on Huawei's innovations in ensuring high availability, linear scalability, rapid recovery, and fault tolerance in large-scale AI model training and inference systems [3][25]. Group 1: High Availability of Super Nodes - AI training and inference require continuous operation, similar to an emergency room, where each computer in the cluster has a backup to take over in case of failure, ensuring uninterrupted tasks [5][6]. - Huawei's CloudMatrix 384 super node employs a fault tolerance strategy that includes system-level, business-level, and operational-level fault management to convert faults into manageable issues [5][6]. Group 2: Linear Scalability - The ideal scenario for computing power is linear scalability, where 100 computers should provide 100 times the power of one. Huawei's task distribution algorithms ensure efficient collaboration among computers, enhancing performance as the number of machines increases [8]. - Key technologies such as TACO, NSF, NB, and AICT have been developed to improve the linearity of training large models, achieving linearity rates of 96% and above in various configurations [8]. Group 3: Rapid Recovery of Training - The system can quickly recover from failures during training by automatically saving progress, allowing it to resume from the last checkpoint rather than starting over [10][12]. - Innovations like process-level rescheduling and online recovery techniques have reduced recovery times to under 3 minutes and even as low as 30 seconds in some cases [12]. Group 4: Fault Tolerance in MoE Model Inference - The article outlines a three-tier fault tolerance strategy for large-scale MoE model inference, which minimizes user impact during hardware failures [14][15]. - Techniques such as instance-level rapid restart and token-level retries have significantly reduced recovery times from 20 minutes to as low as 5 minutes [15]. Group 5: Fault Management and Diagnostic Capabilities - A real-time monitoring system continuously checks the health of each computer in the cluster, allowing for quick identification and resolution of issues [16]. - Huawei's comprehensive fault management solution includes capabilities for error detection, isolation, and recovery, enhancing the reliability of the computing cluster [16]. Group 6: Simulation and Modeling - Before training complex AI models, the computing cluster can simulate various scenarios in a virtual environment to identify potential bottlenecks and optimize performance [19][20]. - The introduction of a Markov modeling simulation platform allows for efficient resource allocation and performance tuning, improving throughput and reducing communication delays [20][21]. Group 7: Framework Migration - Huawei's MindSpore framework has rapidly evolved since its open-source launch, providing tools for seamless migration from other frameworks and enhancing execution efficiency [23]. - The framework supports one-click deployment for large models, significantly improving inference performance [23]. Group 8: Future Outlook - The article concludes that the evolution of computing infrastructure will follow a collaborative path between algorithms, computing power, and engineering capabilities, potentially creating a closed loop of innovation driven by application demands [25].

HUAWEI X HUXIU 在通往通用人工智能（AGI）的路上，如何像其他领域一样实现弯道超车，是业界绕不开的 话题。 在过去的十余年时间里，各项单点技术飞速演进，但随着单点技术演进的边际效应递减和系 统复杂度的提升，系统性能的天花板逐步从单点技术的上限演变成系统工程上限：单点优势 越来越像是精致的零件，提升空间有限；但采用系统工程创新，各个部分完美配合、高效协 同，实现整个系统的效能最优，才有更积极的现实意义。 如何在发挥单点技术优势的同时，以整体视角重新构建路径，通过对复杂系统的极致把控与 再组织、找到新的突破可能？解决这个看似不可能的问题，就有望为我们独立引领最前沿技 术发展创造条件。 近期，虎嗅将推出《华为技术披露集》系列内容，通过一系列技术报告，首次全面详述相关 技术细节，为业界提供参考价值。 我们期待通过本系列内容，携手更多伙伴共同构建开放协作的生态系统，助力昇腾生态在中 国的蓬勃发展。 《华为技术披露集》系列 VOL.13 ：万卡集群 你是否注意到，现在的 AI 越来越 "聪明" 了？能写小说、做翻译、甚至帮医生看 CT 片，这 些能力背后离不开一个默默工作的 "超级大脑工厂"——AI 算力集 ...

Core Viewpoint - The article discusses the advancements in AI computing power clusters, highlighting how they enable the training and inference of large AI models through innovative technologies and fault tolerance mechanisms [1][24]. Group 1: Supernode High Availability - AI training and inference require continuous operation, with each computer in the cluster having a backup to ensure seamless task execution during failures [3][4]. - Huawei's CloudMatrix 384 supernode employs a fault tolerance strategy that includes system-level, business-level, and operational-level fault tolerance to maintain high efficiency [3][4]. Group 2: Cluster Linearity - The ideal scenario for computing power clusters is linear scalability, where 100 computers provide 100 times the power of one [6]. - Huawei's task distribution algorithms ensure that each computer operates efficiently, akin to an orchestra, preventing chaos during large-scale model training [6][8]. Group 3: Rapid Recovery for Large-Scale Training - The system can automatically record training progress, allowing for quick recovery from faults without starting over, significantly reducing downtime [10][11]. - Innovations such as process-level rescheduling and online recovery techniques have been developed to minimize recovery times to under 3 minutes [11][15]. Group 4: Fault Management and Diagnostic Capabilities - A real-time monitoring system continuously checks the health of each computer in the cluster, enabling quick identification and resolution of issues [17]. - Huawei's comprehensive fault management solution includes capabilities for error detection, isolation, and recovery, enhancing overall reliability [17][18]. Group 5: Simulation and Modeling - Before actual training, the computing cluster can simulate scenarios in a "digital wind tunnel" to identify potential bottlenecks and optimize performance [19][20]. - The Markov modeling simulation platform allows for multi-dimensional analysis and performance tuning, ensuring efficient resource allocation [19][20]. Group 6: Framework Migration - Huawei's MindSpore framework supports seamless migration from other frameworks, covering over 90% of PyTorch interfaces, enhancing developer accessibility [22]. - The framework also facilitates quick deployment of large models, improving inference performance through integration with mainstream ecosystems [22]. Group 7: Summary and Outlook - Huawei's innovations address various aspects of computing power clusters, including high availability, linearity, rapid recovery, fault tolerance, diagnostic capabilities, simulation, and framework migration [24]. - The future of computing infrastructure is expected to evolve through a collaborative cycle of application demand, hardware innovation, and engineering feedback, leading to specialized computing solutions [24].

Core Viewpoint - The article discusses the advancements in AI computing power clusters, highlighting their critical role in supporting large-scale AI models and ensuring high availability, fault tolerance, and efficient resource management [2][4][39]. Group 1: High Availability of Super Nodes - AI training and inference require continuous operation, similar to an emergency system in hospitals, where each computer in the cluster has a backup to take over in case of failure, ensuring uninterrupted tasks [6][5]. - Huawei's CloudMatrix 384 super node employs a fault tolerance scheme that includes system-level, business-level, and operational-level fault tolerance, transforming faults into manageable issues [7][8]. Group 2: Cluster Linearity - The ideal scenario for computing power clusters is linear scalability, where the total power of 100 computers should be 100 times that of one, achieved through precise task allocation algorithms [10]. - Huawei's team has developed key technologies to enhance training linearity for large models, achieving linearity rates of 96% for the Pangu Ultra 135B model with 4K cards [11][13]. Group 3: Rapid Recovery in Large-Scale Training - When training with thousands of computing units, the system can automatically save progress, allowing for quick recovery from faults without starting over, significantly reducing downtime [14][15]. - Innovations such as process-level rescheduling and online recovery techniques have been introduced to minimize recovery times to under 3 minutes and even 30 seconds for specific faults [16][20]. Group 4: Fault Management and Diagnosis - A real-time monitoring system continuously checks the health of each computer in the cluster, enabling quick identification and resolution of issues before they escalate [24][26]. - Huawei has developed a comprehensive fault management framework that includes capabilities for error detection, isolation, and recovery, enhancing the reliability of the computing infrastructure [24][28]. Group 5: Simulation and Modeling - Before deploying complex AI models, the computing cluster can simulate scenarios in a virtual environment to identify potential bottlenecks and optimize resource allocation [29][30]. - The introduction of a Markov modeling simulation platform allows for multi-dimensional analysis and performance prediction, improving resource efficiency and system stability [30][31]. Group 6: Framework Migration - Huawei's MindSpore framework has rapidly evolved since its open-source launch, providing tools for seamless migration from other frameworks and enhancing performance during training and inference [37][38]. - The framework supports a wide range of applications, enabling quick deployment of large models and improving inference capabilities [38][39].