Core Insights - The emergence of "Agentic AI" is gaining attention in the tech industry, with predictions that 2025 will be the "Year of AI Agents" [1][9] - Concerns have been raised about the actual capabilities and applicability of Agentic AI, with many projects potentially falling into the trap of concept capitalization rather than delivering real value [1][2] Group 1: Current State of Agentic AI - Gartner predicts that by the end of 2027, over 40% of Agentic AI projects will be canceled due to rising costs, unclear business value, or insufficient risk control [1][10] - A survey by Gartner revealed that 19% of organizations have made significant investments in Agentic AI, while 42% have made conservative investments, and 31% are uncertain or waiting [2] Group 2: Misrepresentation and Challenges - There is a trend of "agent washing," where existing AI tools are rebranded as Agentic AI without providing true agent capabilities; only about 130 out of thousands of vendors actually offer genuine agent functions [2][3] - Most current Agentic AI solutions lack clear business value or return on investment (ROI), as they are not mature enough to achieve complex business goals [3][4] Group 3: Performance Evaluation - Research from Carnegie Mellon University indicates that AI agents have significant gaps in their ability to replace human workers in real-world tasks, with the best-performing model, Gemini 2.5 Pro, achieving only a 30.3% success rate in task completion [6][7] - In a separate evaluation for customer relationship management (CRM) scenarios, leading models showed limited performance, with single-turn interactions averaging a 58% success rate, dropping to around 35% in multi-turn interactions [8] Group 4: Industry Reactions and Future Outlook - Companies like Klarna have experienced setbacks with AI tools, leading to a return to human employees for customer service due to quality issues [9] - Despite current challenges, Gartner remains optimistic about the long-term potential of Agentic AI, forecasting that by 2028, at least 15% of daily work decisions will be made by AI agents [10]

Core Viewpoint - The article discusses the development of KRIS-Bench, a benchmark for evaluating the reasoning capabilities of image editing models, focusing on the structured knowledge acquisition process similar to human learning [2][3][16]. Group 1: KRIS-Bench Overview - KRIS-Bench is a collaborative effort involving multiple prestigious institutions aimed at assessing AI's reasoning abilities in image editing [2]. - The benchmark categorizes knowledge into three types: Factual Knowledge, Conceptual Knowledge, and Procedural Knowledge, allowing AI to face progressively complex editing challenges [4][8]. - It features 7 reasoning dimensions and 22 typical editing tasks, ranging from basic to advanced difficulty levels [6]. Group 2: Evaluation Metrics - KRIS-Bench introduces a four-dimensional automated evaluation system to score editing outputs: Visual Consistency, Visual Quality, Instruction Following, and Knowledge Plausibility [10][11][13]. - The evaluation process includes a total of 1,267 image-instruction pairs, meticulously curated by experts to ensure diverse data sources [12]. Group 3: Model Performance Insights - The benchmark tests 10 models, including 3 closed-source and 7 open-source models, revealing performance gaps particularly in procedural reasoning and natural science tasks [14][16]. - Closed-source models like GPT-Image-1 lead in performance, while open-source models like BAGEL-Think show improvements in knowledge plausibility through enhanced reasoning processes [17].

Core Viewpoint - The article discusses the development of KRIS-Bench, a benchmark for evaluating the reasoning capabilities of image editing models, focusing on the structured knowledge acquisition process similar to human learning [2][3][4]. Group 1: Knowledge Structure - KRIS-Bench is designed to assess AI's knowledge structure through three categories: Factual Knowledge, Conceptual Knowledge, and Procedural Knowledge, allowing for a progressive challenge in image editing tasks [4][8]. - The benchmark includes 7 reasoning dimensions and 22 typical editing tasks, ranging from basic to advanced difficulty levels, covering a wide spectrum of challenges [6]. Group 2: Evaluation Metrics - KRIS-Bench introduces a four-dimensional automated evaluation system to score editing outputs, which includes Visual Consistency, Visual Quality, Instruction Following, and Knowledge Plausibility [11][13]. - The evaluation process involves a total of 1,267 image-instruction pairs, meticulously curated by an expert team to ensure diverse data sources and prevent model exploitation [12]. Group 3: Model Performance - The benchmark evaluates 10 models, including 3 closed-source and 7 open-source models, revealing that closed-source models like GPT-Image-1 outperform open-source counterparts in knowledge plausibility [14][18]. - Despite some models showing improvement in factual knowledge tasks, many still struggle with procedural reasoning and complex scientific tasks, indicating a significant gap in deep reasoning capabilities [18].

Core Viewpoint - The MMAR benchmark reveals that most AI models struggle significantly with complex audio reasoning tasks, indicating a gap in their practical applicability in real-world scenarios [1][9][18]. Summary by Sections MMAR Benchmark Overview - MMAR stands for "A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix," consisting of 1000 high-quality audio understanding questions that require multi-step reasoning capabilities [2][3]. Difficulty of MMAR - The benchmark includes questions that assess various reasoning levels, such as signal, perception, semantic, and cultural understanding, with tasks requiring complex reasoning skills and domain-specific knowledge [6][9]. Model Performance - A total of 30 audio-related models were tested, with the best open-source model, Qwen-2.5-Omni, achieving an average accuracy of only 56.7%, while the closed-source model Gemini 2.0 Flash led with 65.6% [11][18]. - Most open-source models performed close to random guessing, particularly in music-related tasks, highlighting significant challenges in recognizing deeper audio information [12][18]. Error Analysis - The primary errors identified in the models included perceptual errors (37%), reasoning errors (20%), knowledge gaps (9%), and other errors (34%), indicating that current AI models face both auditory and cognitive challenges [19]. Future Outlook - The research emphasizes the need for collaboration in data and algorithm innovation to improve audio reasoning capabilities in AI, with a hope for future models that can truly understand audio content and context [20][21].

Core Viewpoint - The recent research from Wharton School and other institutions reveals that the "direct answer" prompt favored by Ultraman significantly reduces model accuracy [1][9]. Group 1: CoT Prompt Findings - Adding Chain of Thought (CoT) commands in prompts does not enhance reasoning models and increases time and computational costs [2][6]. - For reasoning models, the accuracy improvement from CoT is minimal, with o3-mini showing only a 4.1% increase, while time consumption rose by 80% [6][23]. - Non-reasoning models show mixed results with CoT prompts, necessitating careful consideration of benefits versus costs [7][12]. Group 2: Experimental Setup - The research utilized the GPQA Diamond dataset, which includes graduate-level expert reasoning questions, to test various reasoning and non-reasoning models under different conditions [5][9]. - Each model was tested in three experimental environments: forced reasoning, direct answer, and default [10][11]. Group 3: Performance Metrics - Four metrics were used to evaluate the models: overall results, 100% accuracy, 90% accuracy, and 51% accuracy [12][19]. - For non-reasoning models, CoT prompts improved average scores and the "51% correct" metric, with Gemini Flash 2.0 showing the most significant improvement [12][13]. - However, in the 100% and 90% accuracy metrics, the inclusion of CoT prompts led to declines in performance for some models [14][20]. Group 4: Conclusion on CoT Usage - The study indicates that while CoT can improve overall accuracy, it also increases answer instability [15][22]. - For models like o3-mini and o4-mini, the performance gain from using CoT prompts is minimal, and for Gemini 2.5 Flash, all metrics declined [20][21]. - Default settings of models are suggested to be effective for users, as many advanced models already incorporate reasoning processes internally [25].

Core Insights - The article discusses the comprehensive evaluation of large language models (LLMs) for medical tasks, highlighting that DeepSeek R1 achieved a 66% win rate, outperforming other models in a clinical context [1][7][24]. Evaluation Framework - A comprehensive assessment framework named MedHELM was developed, consisting of 35 benchmark tests covering 22 subcategories of medical tasks [12][20]. - The classification system was validated by 29 practicing clinicians from 14 medical specialties, ensuring its relevance to real-world clinical activities [4][17]. Model Performance - DeepSeek R1 led the evaluation with a 66% win rate and a macro average score of 0.75, indicating its superior performance across the benchmark tests [7][24]. - Other notable models included o3-mini with a 64% win rate and Claude 3.7 Sonnet with a 64% win rate, while models like Gemini 1.5 Pro ranked lowest with a 24% win rate [26][27]. Benchmark Testing - The evaluation included 17 existing benchmarks and 13 newly developed tests, with 12 of the new tests based on real electronic health record data [21][20]. - The models showed varying performance across different task categories, with higher scores in clinical case generation and patient communication tasks compared to structured reasoning tasks [32]. Cost-Effectiveness Analysis - A cost analysis was conducted based on the token consumption during the evaluation, revealing that non-reasoning models like GPT-4o mini had lower costs compared to reasoning models like DeepSeek R1 [38][39]. - The analysis indicated that models like Claude 3.5 Sonnet and Claude 3.7 Sonnet provided good value for their performance at lower costs [39].

Core Viewpoint - OpenAI has announced a significant upgrade to ChatGPT's image generation capabilities, utilizing the GPT-4o model for native image creation and editing, which was previously limited to text generation [1][2]. Group 1: Image Generation Features - The GPT-4o model can now create and modify images and photos, marking its first major upgrade in over a year [1]. - This new image generation feature is available to users subscribed to the Pro plan at $200 per month, with plans to extend access to Plus and free users soon [1]. - The image output time for GPT-4o is slightly longer than that of the DALL-E 3 model, but it produces more accurate and detailed images [2]. Group 2: Editing Capabilities - GPT-4o can edit existing images, including those with people, allowing for transformations and "fixes" to foreground and background details [3]. - OpenAI has trained GPT-4o using publicly available data and proprietary data obtained through partnerships with companies like Shutterstock [3]. Group 3: Intellectual Property and Data Usage - OpenAI respects artists' rights and has policies in place to prevent the generation of images that directly mimic the works of living artists [3]. - The company provides a form for creators to request the removal of their works from the training dataset and respects requests to prevent web crawlers from collecting training data [4]. Group 4: Competitive Landscape - The upgrade follows Google's introduction of a similar image output feature in its Gemini 2.0 Flash model, which has faced criticism for lacking protective measures against copyright infringement [4].

作者：Cage、Yongxin、Siqi 编辑：Siqi DeepSeek R1 催化 了 reasoning model 的竞争：在过去的一个月里，头部 AI labs 已经发布了三个 SOTA reasoning models：OpenAI 的 o3-mini 和deep research， xAI 的 Grok 3 和 Anthropic 的 Claude 3.7 Sonnet。 随着头部 Al labs 先后释出自己的 reasoning model，新范式的第一轮竞赛暂时告一段落。 各家 reasoning model 各有长板，但都没有拉开大的领先优势：OpenAI 和 xAI 有着最强的 base model 和 竞赛解题能力，Anthropic 更关注真实世界的工程问题，Claude 3.7 Sonnet 的混合推理模型可能会成为 之后各家发布新模型的标准操作。 在这一波新模型密集发布后的间隙，我们对已有的 reasoning models 发布进行了总结梳理，除了平 行比较各些模型的实际能力和长板外，更重要的目标是识别出本轮发布中的关键信号。 整体上，我们还处于 RL Scaling 的早期 ...

2024 年 12 月的全球 AI 大事记。 文丨贺乾明 编辑丨程曼祺 2024 年 12 月的 AI 月报，你会看到： OpenAI、Google 发布新模型，中国的 DeepSeek 也抢到了风头 GPT-5 训练遇阻的更多细节 强化学习的重要性持续提升 至少有三个团队推出了世界模型 Google 霸占大模型竞技场前三 中国公司在开源社区存在感大涨 博通帮大公司自研 AI 芯片，市值破万亿美元 OpenAI 正式启动转型营利公司 20+ AI 公司获 5000 万美元以上投资，有 2 家中国公司 大模型的幻觉并不是一无是处 以下是我们第 2 期 AI 月报，欢迎大家在留言区补充我们没有提到的重要进展。 技术｜10 亿美元没训出 GPT-5，新版 Scaling Laws 初步证明可行，多款世界模型亮相 GPT-5 训练遇阻的更多细节 OpenAI 训练 GPT-5（代号 Orion）遇阻，是大模型能力提升放缓的重要证据。12 月，多家媒体提供了更多的细 节： 2023 年 4 月推出 GPT-4 后，OpenAI 一直在开发 GPT-5，已经持续 20 个月。OpenAI 看到过乐观信号：24 年 4 月 ...