梁文锋署名DeepSeek新论文：公开V3大模型降本方法

Core Viewpoint - The article discusses the latest advancements in DeepSeek-V3, focusing on how it overcomes hardware bottlenecks in training large models through four innovative technologies [1][2]. Group 1: Innovations in DeepSeek-V3 - DeepSeek-V3 achieves significant training efficiency using only 2048 H800 GPUs, comparable to systems with thousands of GPUs, through memory optimization and multi-head latent attention (MLA) [2][14]. - The memory optimization reduces the key-value cache (KV Cache) size to 70 KB per token, which is 1/7 to 1/4 of traditional methods, alleviating memory pressure especially for long text processing [15][20]. - The model employs a mixture of experts (MoE) and FP8 low-precision training, activating only 37 billion parameters out of 671 billion during training, resulting in a training cost that is 1/10 of dense models like Llama-3.1 [17][18]. Group 2: Communication and Inference Acceleration - DeepSeek-V3 utilizes a multi-plane fat-tree network design to optimize communication, reducing costs by 40% and latency by 30%, supporting scalability to thousands of GPUs [20][21]. - The model implements dual-pipeline execution for attention calculations and expert communication, enhancing throughput by nearly 100% [22]. - Multi-token prediction (MTP) allows the model to generate multiple tokens simultaneously, increasing generation speed by 1.8 times while maintaining an accuracy of 80%-90% [24][25]. Group 3: Future Hardware Expectations - The article outlines five dimensions for future AI hardware improvements, transitioning from passive adaptation to proactive design [28]. - Recommendations include enhancing low-precision computation support, integrating communication frameworks, optimizing network topologies, improving memory systems, and ensuring robustness against failures [30][33][37][40].