Core Viewpoint - Huawei's CloudMatrix 384 represents a next-generation AI data center architecture designed to meet the increasing demands of large-scale AI workloads, featuring a fully interconnected hardware design that integrates 384 Ascend 910C NPUs and 192 Kunpeng CPUs, facilitating dynamic resource pooling and efficient memory management [6][55]. Summary by Sections Introduction to CloudMatrix - CloudMatrix is introduced as a new AI data center architecture aimed at reshaping AI infrastructure, with CloudMatrix 384 being its first production-level implementation optimized for large-scale AI workloads [6][55]. Features of CloudMatrix 384 - CloudMatrix 384 is characterized by high density, speed, and efficiency, achieved through comprehensive architectural innovations that lead to superior performance in computing, interconnect bandwidth, and memory bandwidth [2][3]. - The architecture allows for direct full-node communication via a unified bus (UB), enabling dynamic pooling and unified access to computing, memory, and network resources, which is particularly beneficial for communication-intensive operations [3][7]. Architectural Innovations - The architecture supports four foundational capabilities: scalable communication for tensor and expert parallelism, flexible heterogeneous workload resource combinations, a unified infrastructure for mixed workloads, and memory-level storage through decomposed memory pools [8][9][10]. Hardware Components - The core of CloudMatrix 384 is the Ascend 910C chip, which features a dual-chip package providing a total throughput of up to 752 TFLOPS and high memory bandwidth [17][18]. - Each computing node integrates multiple NPUs and CPUs, connected through a high-bandwidth UB network, ensuring low latency and high performance [22][24]. Software Stack - Huawei has developed a comprehensive software ecosystem for the Ascend NPUs, known as CANN, which facilitates efficient integration with major AI frameworks like PyTorch and TensorFlow [27][33]. Future Directions - Future enhancements for CloudMatrix 384 include integrating VPC and RDMA networks, expanding to larger supernode configurations, and pursuing finer-grained resource decomposition and pooling [58]. - The architecture is expected to evolve to support increasingly diverse AI workloads, including specialized accelerators for various tasks, enhancing flexibility and efficiency [47][48]. Performance Evaluation - CloudMatrix-Infer, a service solution built on CloudMatrix 384, has demonstrated exceptional throughput and low latency in processing tokens during inference, outperforming leading frameworks [57]. Conclusion - Overall, Huawei's CloudMatrix is positioned as an efficient, scalable, and performance-optimized platform for deploying large-scale AI workloads, setting a benchmark for future AI data center infrastructures [55][58].

Core Viewpoint - Nvidia's CEO Jensen Huang indicated that future forecasts will exclude the Chinese market, yet the significance of China to Nvidia remains critical, as evidenced by the emphasis on Huawei as a competitive threat [3] Group 1: Nvidia's Strategy in China - Nvidia is developing a new generation of chips for the Chinese market, based on the GB202 GPU architecture, with plans to launch these new processors as early as July 2024 [3] - The new chips will include two models, referred to as B20 and B40/B30, which may be marketed as variants of the RTX 6000 series to obscure their Blackwell lineage [4] - Recent U.S. export controls have imposed restrictions on memory bandwidth and interconnect speed, leading to the use of GDDR memory in the new chips instead of HBM memory [4] Group 2: Chip Specifications - The B20 chip will utilize Nvidia's ConnectX-8 for interconnect functionality, optimized for small-scale clusters with 8 to 16 cards, primarily for inference tasks [6] - The B30/B40 models will support NVLink interconnect but at reduced speeds compared to standard specifications, with expected bandwidth similar to the H20's 900Gbps [7] - Memory configurations for the new chips are anticipated to include 24GB, 36GB, and 48GB, with the 48GB option being the most likely [8] Group 3: Market Demand and Pricing - The new chips are expected to be priced between $6,500 and $8,000, significantly lower than the H20's price range of $10,000 to $12,000, which may drive sustained customer demand [9] - Full server configurations with these new chips are estimated to cost between $80,000 and $100,000, depending on the connectivity options [9] Group 4: Customer Interest and Market Dynamics - Major Chinese tech companies have shown varying interest in the new chip models, with Tencent favoring the B20 for its cost-effectiveness in inference tasks, while ByteDance is more interested in the B30 and B40 to meet market demand left by the H20's discontinuation [10][11] - Alibaba has not specified a preference for particular models but indicates a strong overall demand for the chips [11] Group 5: Current Situation and Challenges - The true test for Nvidia will come once major Chinese customers receive testing cards, as the evaluation process typically takes about a month before large orders can be placed [12] - Despite Huang's comments, the Chinese market remains a vital revenue source for Nvidia, and competitors like Huawei continue to advance their own R&D efforts [12]