Group 1 - The core idea of the article is the introduction of Xbench, an AI benchmarking tool developed by Sequoia China, which emphasizes the importance of evaluating AI systems based on their practical utility in real-world scenarios rather than just the difficulty of the assessment questions [1][5][6] - Xbench was initiated in late 2022 as an internal tool for tracking and evaluating the capabilities of foundational models, evolving through three major updates, with the first public release planned for May 2025 [5][6] - The dual-track evaluation system of Xbench includes AGI Tracking to assess the upper limits of agent capabilities and Profession Aligned to quantify the practical value of AI systems in real-world applications [6][8] Group 2 - The Evergreen Evaluation Mechanism is a dynamic updating evaluation system designed to avoid the pitfalls of static assessments, which can lead to overfitting and rapid obsolescence [10] - The mechanism aims to regularly evaluate mainstream agent products across various sectors such as human resources, marketing, finance, law, and sales, adapting to the fast-paced evolution of agent applications [10] Group 3 - In the initial phase of testing, significant performance differences were observed among various models in recruitment and marketing tasks, with OpenAI's o3 ranking first and GPT-4o scoring the lowest due to its tendency to provide shorter answers [9] - The evaluation revealed that model size is not the sole determinant of task performance, as demonstrated by the comparable results of Google's DeepMind models [9] - Despite DeepSeek R1's strong performance in mathematical and coding benchmarks, its adaptability issues in search-centric tasks led to lower scores in this evaluation [9]

Core Viewpoint - The traditional AI benchmarking is rapidly losing effectiveness as many models achieve perfect scores, leading to a lack of differentiation and guidance in evaluation [1][2]. Group 1: Introduction of xbench - Sequoia China launched a new AI benchmark test called xbench, aiming to create a more scientific and long-lasting evaluation system that reflects the objective capabilities of AI [2][3]. - xbench is the first benchmark initiated by an investment institution, collaborating with top universities and research institutions, utilizing a dual-track evaluation system and an evergreen evaluation mechanism [2][3]. Group 2: Features of xbench - xbench employs a dual-track evaluation system to track both the theoretical capability limits of models and the practical value of AI systems in real-world applications [3][4]. - The evergreen evaluation mechanism ensures that the testing content is continuously maintained and updated to remain relevant and timely [3][4]. - The initial release includes two core evaluation sets: xbench-ScienceQA and xbench-DeepSearch, along with a ranking of major products in these fields [4][10]. Group 3: Addressing Core Issues - Sequoia China identified two core issues: the relationship between model capabilities and actual AI utility, and the loss of comparability in AI capabilities over time due to frequent updates in the question bank [5][6]. - xbench aims to break away from conventional thinking by developing novel task settings and evaluation methods that align with real-world applications [6][7]. Group 4: Dynamic Evaluation Mechanism - xbench plans to establish a dynamic evaluation mechanism that collects live data from real business scenarios, inviting industry experts to help build and maintain the evaluation sets [9][8]. - The design includes horizontally comparable capability metrics to observe development speed and key breakthroughs over time, aiding in determining when an agent can take over existing business processes [9][8]. Group 5: Community Engagement - xbench encourages community participation, allowing developers and researchers to use the latest evaluation sets to validate their products and contribute to the development of industry-specific standards [11][10].

Core Viewpoint - The rapid development of foundational models and the scaling application of AI agents have led to challenges in accurately reflecting the objective capabilities of AI systems through benchmark tests, necessitating the creation of a more scientific and sustainable evaluation system to guide AI technology breakthroughs and product iterations [1][2]. Group 1: Introduction of xbench - Sequoia China announced the launch of a new AI benchmark test called xbench, which is the first benchmark initiated by an investment institution in collaboration with top universities and research institutions, utilizing a dual-track evaluation system and evergreen evaluation mechanism [2][4]. - xbench aims to assess and enhance the capabilities of AI systems while quantifying their utility value in real-world scenarios, capturing key breakthroughs in agent products over time [2][4]. Group 2: Features of xbench - xbench employs a dual-track evaluation system that constructs a multidimensional dataset to track both the theoretical capability limits of models and the practical value of agents [4][5]. - The evaluation tasks are divided into two complementary main lines: assessing the upper limits of AI system capabilities and quantifying their utility value in real-world applications [4][6]. - An evergreen evaluation mechanism is adopted to ensure the timeliness and relevance of the testing content by continuously maintaining and dynamically updating the test materials [4][10]. Group 3: Addressing Core Issues - Sequoia China identified two core issues with existing evaluation methods: the relationship between model capabilities and actual AI utility, and the loss of comparability in AI capabilities over time due to frequent updates of test materials [6][7]. - To address these issues, xbench proposes innovative task settings and evaluation methods aligned with real-world applications, introducing a dual-track system that includes AGI tracking and profession-aligned assessments [7][8]. Group 4: Initial Assessment Sets - The first release of xbench includes two core assessment sets: xbench-ScienceQA for scientific question answering and xbench-DeepSearch for deep search capabilities, along with a comprehensive ranking of major products in these fields [8][11]. - xbench has been used internally by Sequoia China for tracking and evaluating foundational model capabilities over the past two years and is now publicly available for the AI community [8][11]. Group 5: Community Collaboration - Sequoia China encourages community collaboration in building and publishing specific industry standards for profession-aligned xbench, inviting developers and researchers to contribute to the ongoing development and maintenance of evaluation updates [11][13].