HUAWEI X HUXIU 在通往通用人工智能（AGI）的路上，如何像其他领域一样实现弯道超车，是业界绕不开的 话题。 在过去的十余年时间里，各项单点技术飞速演进，但随着单点技术演进的边际效应递减和系 统复杂度的提升，系统性能的天花板逐步从单点技术的上限演变成系统工程上限：单点优势 越来越像是精致的零件，提升空间有限；但采用系统工程创新，各个部分完美配合、高效协 同，实现整个系统的效能最优，才有更积极的现实意义。 如何在发挥单点技术优势的同时，以整体视角重新构建路径，通过对复杂系统的极致把控与 再组织、找到新的突破可能？解决这个看似不可能的问题，就有望为我们独立引领最前沿技 术发展创造条件。 近期，虎嗅将推出《华为技术披露集》系列内容，通过一系列技术报告，首次全面详述相关 技术细节，为业界提供参考价值。 我们期待通过本系列内容，携手更多伙伴共同构建开放协作的生态系统，助力昇腾生态在中 国的蓬勃发展。 《华为技术披露集》系列 VOL.13 ：万卡集群 你是否注意到，现在的 AI 越来越 "聪明" 了？能写小说、做翻译、甚至帮医生看 CT 片，这 些能力背后离不开一个默默工作的 "超级大脑工厂"——AI 算力集 ...

Core Viewpoint - The article discusses the advancements in AI computing power clusters, highlighting how they enable the training and inference of large AI models through innovative technologies and fault tolerance mechanisms [1][24]. Group 1: Supernode High Availability - AI training and inference require continuous operation, with each computer in the cluster having a backup to ensure seamless task execution during failures [3][4]. - Huawei's CloudMatrix 384 supernode employs a fault tolerance strategy that includes system-level, business-level, and operational-level fault tolerance to maintain high efficiency [3][4]. Group 2: Cluster Linearity - The ideal scenario for computing power clusters is linear scalability, where 100 computers provide 100 times the power of one [6]. - Huawei's task distribution algorithms ensure that each computer operates efficiently, akin to an orchestra, preventing chaos during large-scale model training [6][8]. Group 3: Rapid Recovery for Large-Scale Training - The system can automatically record training progress, allowing for quick recovery from faults without starting over, significantly reducing downtime [10][11]. - Innovations such as process-level rescheduling and online recovery techniques have been developed to minimize recovery times to under 3 minutes [11][15]. Group 4: Fault Management and Diagnostic Capabilities - A real-time monitoring system continuously checks the health of each computer in the cluster, enabling quick identification and resolution of issues [17]. - Huawei's comprehensive fault management solution includes capabilities for error detection, isolation, and recovery, enhancing overall reliability [17][18]. Group 5: Simulation and Modeling - Before actual training, the computing cluster can simulate scenarios in a "digital wind tunnel" to identify potential bottlenecks and optimize performance [19][20]. - The Markov modeling simulation platform allows for multi-dimensional analysis and performance tuning, ensuring efficient resource allocation [19][20]. Group 6: Framework Migration - Huawei's MindSpore framework supports seamless migration from other frameworks, covering over 90% of PyTorch interfaces, enhancing developer accessibility [22]. - The framework also facilitates quick deployment of large models, improving inference performance through integration with mainstream ecosystems [22]. Group 7: Summary and Outlook - Huawei's innovations address various aspects of computing power clusters, including high availability, linearity, rapid recovery, fault tolerance, diagnostic capabilities, simulation, and framework migration [24]. - The future of computing infrastructure is expected to evolve through a collaborative cycle of application demand, hardware innovation, and engineering feedback, leading to specialized computing solutions [24].

Core Viewpoint - The article discusses the advancements in AI computing power clusters, highlighting their critical role in supporting large-scale AI models and ensuring high availability, fault tolerance, and efficient resource management [2][4][39]. Group 1: High Availability of Super Nodes - AI training and inference require continuous operation, similar to an emergency system in hospitals, where each computer in the cluster has a backup to take over in case of failure, ensuring uninterrupted tasks [6][5]. - Huawei's CloudMatrix 384 super node employs a fault tolerance scheme that includes system-level, business-level, and operational-level fault tolerance, transforming faults into manageable issues [7][8]. Group 2: Cluster Linearity - The ideal scenario for computing power clusters is linear scalability, where the total power of 100 computers should be 100 times that of one, achieved through precise task allocation algorithms [10]. - Huawei's team has developed key technologies to enhance training linearity for large models, achieving linearity rates of 96% for the Pangu Ultra 135B model with 4K cards [11][13]. Group 3: Rapid Recovery in Large-Scale Training - When training with thousands of computing units, the system can automatically save progress, allowing for quick recovery from faults without starting over, significantly reducing downtime [14][15]. - Innovations such as process-level rescheduling and online recovery techniques have been introduced to minimize recovery times to under 3 minutes and even 30 seconds for specific faults [16][20]. Group 4: Fault Management and Diagnosis - A real-time monitoring system continuously checks the health of each computer in the cluster, enabling quick identification and resolution of issues before they escalate [24][26]. - Huawei has developed a comprehensive fault management framework that includes capabilities for error detection, isolation, and recovery, enhancing the reliability of the computing infrastructure [24][28]. Group 5: Simulation and Modeling - Before deploying complex AI models, the computing cluster can simulate scenarios in a virtual environment to identify potential bottlenecks and optimize resource allocation [29][30]. - The introduction of a Markov modeling simulation platform allows for multi-dimensional analysis and performance prediction, improving resource efficiency and system stability [30][31]. Group 6: Framework Migration - Huawei's MindSpore framework has rapidly evolved since its open-source launch, providing tools for seamless migration from other frameworks and enhancing performance during training and inference [37][38]. - The framework supports a wide range of applications, enabling quick deployment of large models and improving inference capabilities [38][39].