中国电信研究院取得大模型分布式推理技术突破

Core Insights - China Telecom Research Institute, in collaboration with Peking University, has achieved a significant technological breakthrough in distributed inference optimization for large models, addressing the core contradiction between inference efficiency and hardware costs [1][2] Group 1: Technical Achievements - A high-efficiency, low-cost enterprise-level LLM inference optimization solution has been developed, covering major application scenarios for large model inference [1] - In multi-task scenarios, a scheduling algorithm was developed to reduce average end-to-end latency by 40% and decrease short request first token latency and decoding latency by 75% in mixed request lengths from 1k to 32k [2] - An improved low-bit quantization algorithm was implemented to compress model weights while maintaining accuracy, reducing the minimum deployment unit from 6 A800 units to 1, resulting in over 80% savings in hardware costs and a 50% increase in inference efficiency [2] Group 2: Practical Applications and Impact - Since its launch earlier this year, the technology has provided API services for over 30 research projects, processing more than 26 billion tokens, demonstrating its feasibility and performance gains in inference efficiency and throughput [3] - The technology has been piloted in various projects within the China Telecom Group and provincial companies, providing a reference for large-scale deployment solutions [3] - Future efforts will focus on further optimizing distributed inference technology and collaborating with industry partners to drive innovation and standardization in inference optimization [3]