Core Insights - The article emphasizes that the Total Cost of Ownership (TCO) is highly dependent on the specific use case, suggesting that TPU is preferable for training and latency-insensitive inference, while GPU is better for prefill and latency-sensitive inference scenarios [3][4][5] - The fundamental difference between the 3D Torus and Switch Fabric (NVSwitch/Fat-tree) interconnect systems lies not in speed but in their assumptions about traffic patterns [4][5] - Google's historical TCO advantage established through TPU has been significantly weakened in the v8 generation [6] TCO Analysis - TPU v7 offers a cost advantage of 45-56% in training scenarios, based on the assumption that TPU's Model FLOPs Utilization (MFU) is 5-10 percentage points higher than that of GPUs [4][16] - In inference scenarios, GPUs (GB200/GB300) outperform TPU v7 by approximately 35-50% during the prefill phase due to their FP4 computational advantage [4][18] - The TCO comparison shows that TPU v8's cost efficiency has decreased, with the TCO ratio dropping from 1.52x for GB200/TPUv7 to 1.23x for VR200/TPUv8p [6] Interconnect Architecture - The 3D Torus architecture assumes predictable and orchestrated communication patterns, maintaining high MFU in large-scale training tasks, while Switch Fabric accommodates uncertain traffic patterns [5][38] - TPU Pods utilize a 3D Torus topology for high bandwidth and low latency communication, with a maximum cluster size limited by the number of OCS ports [31][34] Performance Bottlenecks - In training, the bottleneck typically arises from computational power and scale-out communication bandwidth, while in inference, the prefill phase is limited by computational power and the decode phase is constrained by memory bandwidth [12][22] - The performance requirements differ across training and inference scenarios, with TPU needing FP8 and scale-out bandwidth for training, while GPU requires FP4 and scale-up bandwidth for inference [12][13] Software Optimization - TPU's software optimizations aim to mitigate its inherent weaknesses in handling irregular traffic, transforming unpredictable workloads into stable data flows [46][47] - The introduction of SparseCore in TPU is designed to enhance its capability to handle dynamic all-to-all routing, acknowledging the need for communication-computation decoupling similar to NVSwitch [48] Competitive Landscape - Google TPU v8 adopts a dual-supplier strategy to reduce costs, collaborating with Broadcom and MediaTek for different SKUs, which impacts the overall design and production timeline [49][50] - Nvidia's Rubin architecture aggressively enhances performance and TCO for inference, with significant improvements in FP4 computational power and HBM bandwidth, positioning it as a strong competitor against TPU [51][52]