清华教授翟季冬：Benchmark正在「失效」，智能路由终结大模型选型乱象

Core Insights - The article discusses the "choice paradox" in the AI model and computing power industry, highlighting the challenges users face in selecting appropriate models amidst a plethora of options and varying performance metrics [2][7][10] - It emphasizes that high benchmark scores do not necessarily align with user needs, as different service providers may offer significantly different performance for the same model due to factors like aggressive quantization [8][10][11] - The article introduces AI Ping, a product developed by Qingcheng Jizhi, aimed at providing a systematic evaluation of different models and service providers, thereby helping users make informed decisions [3][12][17] Group 1: Industry Challenges - Users often struggle with the overwhelming number of options and the complexity of selecting the right model, which can lead to inefficiencies and increased costs for enterprises [2][10] - The performance of models can vary widely based on the service provider, with discrepancies in API service throughput and response times affecting user experience [8][9] - The article notes that the choice of model should be tailored to specific tasks, as different models excel in different areas, which complicates the selection process for users [10][11] Group 2: AI Ping and Its Functionality - AI Ping aims to act as a "Yelp for computing power," aggregating performance data and user habits to recommend cost-effective solutions [3][17] - The product's functionality includes both service provider routing and model routing, allowing users to select the best service and model based on their specific needs [13][17] - The development of AI Ping has involved extensive testing of various models and service providers to ensure accurate performance metrics and user satisfaction [14][19] Group 3: Market Dynamics and Future Directions - The article highlights the importance of data aggregation in improving model selection accuracy, which can lead to reduced costs for users and better resource utilization for service providers [3][17] - It discusses the evolving landscape of the AI Infra industry, emphasizing the need for continuous software and hardware integration to meet the growing demands of users [22][30] - The article concludes with a reflection on the future of AI Infra, suggesting that as long as model evolution and computing architecture continue to advance, the demand for AI Infra solutions will persist [26][30]