Industry Overview - The semiconductor industry is experiencing a new storage cycle driven by emerging technologies and AI demand, with historical cycles characterized by demand, capacity, and inventory phases [1][2] - The memory segment, being the second largest in semiconductors, shows greater volatility than the overall industry, with significant market growth expected due to AI [1] Capital Expenditure Projections - DRAM capital expenditure is projected to reach $53.7 billion in 2025, increasing to $61.3 billion in 2026, representing a year-on-year growth of approximately 14% [3] - NAND Flash capital expenditure is expected to be $21.1 billion in 2025, with a slight increase to $22.2 billion in 2026, reflecting a year-on-year growth of about 5% [3] AI Impact on Storage Demand - The introduction of reasoning chains in large language models (LLMs) is significantly increasing data storage needs, with a shift from KB to TB and even EB in storage units [2] - The cost of reasoning in large models has decreased exponentially since the release of ChatGPT-3, which is expected to drive application growth and storage demand [2] - KV Cache is identified as a key mechanism for optimizing reasoning efficiency in large models, further increasing storage requirements [2] Current Industry Focus - Memory manufacturers are shifting focus from pure capacity expansion to upgrading process technologies and developing high-value products like HBM [3] - Current cleanroom space is nearing capacity limits, with only a few manufacturers like Samsung and SK Hynix having limited expansion capabilities [3] Investment Recommendations - Continuous monitoring of memory inventory, pricing data, and the impact of AI computing power on storage chip demand is advised [4]

Core Viewpoint - The article discusses the challenges faced by large language models (LLMs) due to the "memory wall" issue and proposes SSD-based storage offloading technology as a new path for efficient AI model operation [1]. Group 1: Industry Insights and Investment Recommendations - The massive data generated by AI is impacting global data center storage facilities, leading to a focus on KV Cache caching that can offload from GPU memory to CPU and SSD [1]. - The traditional Nearline HDD, which has been a cornerstone for massive data storage, is experiencing supply shortages, prompting a shift towards high-performance, high-cost SSDs, resulting in an "overweight" rating for the industry [1]. Group 2: KV Cache Technology and Its Implications - The growth of KV Cache capacity is exceeding the capabilities of HBM, as it temporarily stores generated tokens to optimize computational efficiency and reduce redundant calculations [2]. - As the demand for larger models and longer sequences increases, the reliance on HBM is becoming a bottleneck, leading to frequent memory overflows and performance issues [2]. Group 3: Technological Developments in Storage Solutions - The industry is exploring tiered caching management technologies for KV Cache, with NVIDIA launching a distributed inference service framework called Dynamo to offload KV Cache from GPU memory to CPU, SSD, and even network storage [3]. - Samsung has proposed an SSD-based storage offloading solution to address the "memory wall" challenge, which can reduce the first token latency by up to 66% and inter-token latency by up to 42% when KV Cache size exceeds HBM or DRAM capacity [3]. Group 4: Market Trends and Supply Chain Dynamics - The demand for AI storage is driving a replacement effect for HDDs, with NAND Flash suppliers accelerating the production of large-capacity Nearline SSDs due to significant supply gaps in the HDD market [4]. - NAND Flash manufacturers are investing in the production of ultra-large capacity Nearline SSDs, such as 122TB and even 245TB models, to meet the growing demand from AI inference applications [4].

Core Insights - The article discusses the importance of KV Cache in enhancing the efficiency of large language models (LLMs) during autoregressive inference, particularly in the context of the Transformer architecture [1][20]. Group 1: Need for KV Cache - KV Cache is essential for storing intermediate computation results, which significantly improves the model's operational efficiency during text generation tasks [1][20]. - In standard Transformer decoding, each new token generation requires attention calculations that involve all previous tokens, leading to high computational complexity [2][6]. Group 2: Working Principle of KV Cache - The core idea of KV Cache is to cache the historical Key (K) and Value (V) matrices, thus avoiding redundant calculations and reducing time complexity from O(n²) to O(n) [4][7]. - The process involves calculating the new Query (Q) matrix and performing attention calculations with the cached K and V matrices, allowing for efficient token generation [4][10]. Group 3: Technical Details of KV Cache - KV Cache typically maintains independent caches for each attention head, with the cache structure dynamically growing until it reaches the model's maximum sequence length [11]. - While KV Cache improves speed, it requires additional memory, with models like GPT-3 consuming approximately 20KB of memory per token, leading to significant memory usage during batch processing [12]. Group 4: Optimization Strategies for KV Cache - Strategies such as Paged KV Cache, dynamic cache management, quantization, and selective caching are employed to enhance the efficiency of KV Cache while managing memory usage [22][18]. Group 5: Code Implementation - The article provides a code example demonstrating the implementation of KV Cache in self-attention mechanisms using PyTorch, highlighting the modifications needed to incorporate caching [14][17]. Group 6: Conclusion - Understanding the workings of KV Cache is crucial for optimizing inference performance in large models and addressing challenges in practical deployment [20].

Core Insights - The article discusses the challenges and advancements in long-context large language models (LLMs), particularly focusing on KV cache optimization methods to enhance computational and memory efficiency [2][6][12]. Group 1: Long-Context LLMs and Their Challenges - Long-context LLMs have become mainstream, significantly improving performance in various applications by supporting context windows of millions of tokens [5][6]. - The ability to handle longer contexts enhances the model's understanding and problem-solving capabilities, especially in complex tasks like debugging and multi-turn dialogues [5][6]. - However, the use of long contexts incurs high costs and significantly reduces inference speed due to computational complexity and storage pressure from KV cache [6][11]. Group 2: Optimization Strategies - Several optimization strategies have been proposed to address the challenges of long-context LLMs, including MInference, which reduces pre-filling latency by an order of magnitude [11][45]. - RetrievalAttention alleviates the memory pressure of KV cache, enabling context inference of up to 128K tokens even on consumer-grade GPUs [11][95]. - The article emphasizes the importance of cross-request optimization, such as Prefix Cache reuse, to improve overall processing efficiency in multi-request scenarios [11][17]. Group 3: SCBench and Benchmarking - SCBench is introduced as a comprehensive benchmarking tool that models the full lifecycle of KV cache in real-world applications, focusing on multi-turn dialogues and enterprise-level document queries [3][25]. - The benchmark includes various tasks to evaluate the model's performance in long-context environments, covering string-level and semantic-level retrieval capabilities [27][28]. Group 4: Dynamic Sparse Attention - The article highlights the dynamic sparsity of attention mechanisms, which can lead to significant computational savings by focusing only on relevant tokens during inference [39][45]. - MInference leverages this dynamic sparsity to achieve up to 10x acceleration in inference tasks, reducing the time required for processing large token inputs [46][51]. - The framework for dynamic sparse attention is designed to optimize both training and inference phases, enhancing overall model efficiency [83][106]. Group 5: Future Directions - Future research may explore the application of dynamic sparsity in long generation tasks and reinforcement learning training phases, aiming to improve efficiency across various stages of model deployment [106][107]. - The community's interest in dynamic sparse attention methods has grown, leading to the emergence of various related works that focus on refining estimation strategies and integrating sparse modeling into training processes [80][81].

Core Viewpoint - Huawei has launched the Unified Cache Manager (UCM), an AI inference memory data management technology aimed at optimizing inference speed, efficiency, and cost in large model inference processes [1][3]. Group 1: UCM Technology Overview - UCM is a KV Cache-centered inference acceleration suite that integrates various caching acceleration algorithms to manage KV Cache memory data generated during inference, thereby expanding the context window for inference [1][3]. - The technology aims to enhance the AI inference experience, improve cost-effectiveness, and accelerate the commercial cycle of AI applications [1][4]. - UCM features a hierarchical adaptive global prefix caching technology that can reduce the latency of the first token by up to 90% [3][6]. Group 2: Industry Application and Impact - In a pilot application with China UnionPay, UCM technology improved large model inference speed by 125 times, allowing for precise identification of customer queries in just 10 seconds [4]. - The financial sector is the first to adopt this technology due to its digital nature and high demands for speed, efficiency, and reliability, making it an ideal testing ground for new AI technologies [4][6]. Group 3: Differentiation and Competitive Advantage - UCM's differentiation lies in its integration of professional storage capabilities, offering a comprehensive lifecycle management mechanism for KV Cache, including preheating, tiering, and elimination [6][7]. - Unlike existing solutions that primarily focus on prefix caching, UCM incorporates a broader range of algorithms, including sparse full-process algorithms and suffix retrieval algorithms, enhancing its reliability and effectiveness [6][7]. - UCM is designed to adapt to various inference scenarios, allowing for smooth optimization across different input and output conditions [6][7]. Group 4: Open Source Initiative and Industry Collaboration - Huawei plans to open source UCM in September, providing a unified interface that can adapt to various inference engines, computing power, and storage systems, promoting collaboration across the industry [7]. - The company aims to address efficiency and cost issues in the AI industry by fostering a collaborative ecosystem among framework vendors, storage providers, and computing power suppliers [7].

Core Insights - The article discusses the challenges and advancements in long-context large language models (LLMs), particularly focusing on KV cache optimization methods to enhance computational efficiency and memory usage [2][3][4]. Long Context LLMs - Long-context LLMs have become mainstream, significantly improving model performance by allowing the integration of extensive contextual information, such as meeting minutes and technical documents [5][6]. - Models like Gemini support context windows of millions of tokens, enhancing performance in applications requiring complex decision-making [5][6]. Challenges in Long Context Usage - The use of long-context LLMs incurs high costs and reduced inference speeds due to two main challenges: computational complexity leading to latency and storage pressure from KV cache [6][11]. - For instance, processing 1 million tokens on an 8B parameter model can take over 30 minutes on an A100 GPU, necessitating multiple GPUs for efficient service [6][11]. Optimization Strategies - Several optimization strategies have been proposed, including MInference, which reduces pre-filling latency by an order of magnitude, and RetrievalAttention, which alleviates KV cache memory pressure [11][12]. - The article emphasizes the importance of cross-request optimization, particularly through prefix cache reuse, to enhance overall processing efficiency [11][17]. KV Cache Lifecycle - The article introduces SCBench, a comprehensive benchmarking tool that models the full lifecycle of KV cache in real-world applications, addressing the need for a holistic approach to optimization [24][25]. - Two common scenarios for KV cache reuse are identified: multi-turn dialogues and enterprise-level document queries, both exhibiting significant context overlap [25]. Performance Evaluation - SCBench includes 12 sub-tasks covering various long-context modeling methods and incorporates four KV cache optimization strategies to assess model performance in practical inference tasks [27]. - The evaluation metrics include string-level and semantic-level context recall, global information understanding, and multi-task processing capabilities [27]. Dynamic Sparse Attention - The article discusses the dynamic sparse attention mechanism, which leverages the inherent sparsity of attention calculations to enhance inference efficiency [40][46]. - MInference 1.0 is introduced as a method that utilizes dynamic sparsity to reduce the number of tokens involved in calculations, achieving up to 10x acceleration in inference tasks [47][50]. Multi-Modal Input Challenges - In multi-modal scenarios, attention mechanisms exhibit pronounced bias characteristics, necessitating adjustments to optimize computational efficiency [55][60]. - The proposed MMInference framework addresses these challenges by employing a two-level attention mechanism to handle both inter-modal and intra-modal attention patterns [63]. Future Directions - The article concludes with a vision for future research, suggesting that dynamic sparsity can enhance efficiency not only in pre-filling and decoding but also in long text extension and generation phases [107][108].