Core Insights - The rapid growth of AI inference applications is significantly increasing the reliance on high-performance memory and tiered storage, with HBM, DRAM, SSD, and HDD playing critical roles in long-context and multimodal inference scenarios [1][2][3] - The overall demand for storage is expected to surge to hundreds of exabytes (EB) as lightweight model deployment drives storage capacity needs [1][3] Group 1: Storage in AI Servers - Storage in AI servers primarily includes HBM, DRAM, and SSD, characterized by decreasing performance, increasing capacity, and decreasing costs [1] - Frequently accessed or mutable data is retained in higher storage tiers, such as CPU/GPU caches, HBM, and dynamic RAM, while infrequently accessed or long-term data is moved to lower storage tiers like SSD and HDD [1] Group 2: Tiered Storage for Efficient Computing - HBM is integrated within GPUs to provide high-bandwidth temporary buffering for weights and activation values, supporting parallel computing and low-latency inference [2] - DRAM serves as system memory, storing intermediate data, batch processing queues, and model I/O, facilitating efficient data transfer between CPU and GPU [2] - Local SSDs are used for real-time loading of model parameters and data, meeting high-frequency read/write needs, while HDDs offer economical large capacity for raw data and historical checkpoints [2] Group 3: Growth Driven by Inference Needs - Memory benefits from long-context and multimodal inference demands, where high bandwidth and large capacity memory reduce access latency and enhance parallel efficiency [3] - For example, the Mooncake project achieved computational efficiency leaps through resource reconstruction, and various upgrades in hardware support high-performance inference in complex models [3] - Based on key assumptions, the storage capacity required for ten Google-level inference applications by 2026 is estimated to be 49EB [3]