2025年08月25日 证券研究报告 | 人工智能行业专题（11） 探究模型能力与应用的进展和边界 行业研究 · 行业专题 互联网 · 互联网II 投资评级：优于大市（维持） 证券分析师：张伦可 证券分析师：陈淑媛 证券分析师：刘子谭 证券分析师：张昊晨 0755-81982651 021-60375431 liuzitan@guosen.com.cn zhanghaochen1@guosen.com.cn zhanglunke@guosen.com.cn chenshuyuan@guosen.com.cn S0980525060001 S0980525010001 S0980521120004 S0980524030003 请务必阅读正文之后的免责声明及其项下所有内容 报告摘要 Ø 风险提示：宏观经济波动风险、广告增长不及预期风险、行业竞争加剧风险、AI技术进展不及预期风险等。 请务必阅读正文之后的免责声明及其项下所有内容 2 Ø 本篇报告主要针对海内外模型发展、探究模型能力与应用的进展和边界。我们认为当前海外模型呈现差异化发展，企业调用考虑性价比。当前 OpenAI在技术路径上相对领先，聚焦强化推理与专业 ...

Model Performance & Benchmarks - GPT5 demonstrates varied performance across different reasoning effort configurations, ranging from frontier levels to GPT-4.1 levels [6] - GPT5 achieves a score of 68 on the artificial intelligence index, setting a new standard [7] - Token usage for GPT5 varies significantly, with high reasoning effort using 82 million tokens compared to minimal reasoning effort using only 3.5 million tokens [8] - LM Arena ranks GPT5 as number one across the board, with an ELO score of 1481, surpassing Gemini 2.5 Pro at 1460 [19][20] - Stage Hand's evaluations indicate GPT5 performs worse than Opus 4.1 in both speed and accuracy for browsing use cases [25] - XAI's Grok 4 outperforms GPT5 in the ARC AGI benchmark [34][51] User Experience & Customization - User feedback indicates a preference for the personality and familiarity of GPT-4.0, even if GPT5 performs better in most ways [2][3] - OpenAI plans to focus on making GPT5 "warmer" to address user concerns about its personality [4] - GPT5 introduces reasoning effort configurations (high, medium, low, minimal) to steer the model's thinking process [6] - GPT5 was launched with a model router to route to the most appropriate flavor size of that model speed of that model depending on the prompt and use case [29] Pricing & Accessibility - GPT5 is priced at $1.25 per million input tokens and $10 per million output tokens [36] - GPT5 is more than five times cheaper than Opus 4.1 and greater than 40% cheaper than Sonnet [39]

Core Viewpoint - Security risk management is not merely a cost center but a value engine for companies to build brand reputation and gain market trust in the AI era [2][4]. Group 1: AI Risks and Security - AI risks have already become a reality, as evidenced by the recent vulnerability in the open-source model tool Ollama, which had an unprotected port [6][12]. - The notion of "exchanging privacy for convenience" is dangerous and can lead to irreversible risks, as AI can reconstruct personal profiles from fragmented data [6][10]. - AI risks are a "new species," and traditional methods are inadequate to address them due to their inherent complexities, such as algorithmic black boxes and model hallucinations [6][12]. - Companies must develop new AI security protection systems that adapt to these unique characteristics [6][12]. Group 2: Strategic Advantages of Security Compliance - Security compliance should be viewed as a strategic advantage rather than a mere compliance action, with companies encouraged to transform compliance requirements into internal risk control indicators [6][12]. - The approach to AI application registration should focus on enhancing risk management capabilities rather than just fulfilling regulatory requirements [6][15]. Group 3: Recommendations for Enterprises - Companies should adopt a mixed strategy of "core closed-source and peripheral open-source" models, using closed-source for sensitive operations and open-source for innovation [7][23]. - To ensure the long-term success of AI initiatives, companies should cultivate a mindset of curiosity, pragmatism, and respect for compliance [7][24]. - A systematic AI security compliance governance framework should be established, integrating risk management into the entire business lifecycle [7][24]. Group 4: Emerging Threats and Defense Mechanisms - "Prompt injection" attacks are akin to social engineering and require multi-dimensional defense mechanisms, including input filtering and sandbox isolation [7][19]. - Companies should implement behavior monitoring and context tracing to enhance security against sophisticated AI attacks [7][19][20]. - The debate between open-source and closed-source models is not binary; companies should choose based on their specific needs and risk tolerance [7][21][23].

Core Insights - The emergence of Agents marks a significant shift in the AI landscape, transitioning from large models as mere tools to self-scheduling intelligent entities [1][2] - The Agent sector is rapidly gaining traction, with a consensus forming around its potential, yet many products struggle to deliver real user value, often repackaging old demands with new technologies [2][3] - The true challenges for Agents lie not in model capabilities but in foundational infrastructure, including controllable operating environments, memory systems, context awareness, and tool utilization [2][3] Group 1: Market Dynamics - The Agent market is characterized by a supply overflow and unclear demand, prompting a need to identify genuine problems and opportunities within this space [2][3] - Successful Agents must evolve from initial Copilot functionalities to fully autonomous systems, leveraging user data and experience to transition effectively [9][19] - Coding is viewed as a critical domain for achieving AGI, with the potential to capture a significant portion of the value in the large model industry [11][25] Group 2: Product Development and User Experience - A successful Agent must create a verifiable data environment, allowing for reinforcement learning from clear rewards, particularly in structured fields like coding [26][27] - The design of AI Native products should consider both human and AI needs, ensuring a dual mechanism that serves both parties effectively [31][32] - User experience metrics, such as task completion rates and user retention, are essential for evaluating an Agent's effectiveness and potential [30][31] Group 3: Business Models and Commercialization - The trend is shifting from cost-based pricing to value-based pricing models, with various innovative approaches emerging, such as charging per action or workflow [36][41] - Future commercial models may include paying for the Agent itself, akin to employment contracts, which could redefine the relationship between users and AI [42][43] - The integration of smart contracts in the Agent ecosystem presents a unique opportunity for establishing economic incentives based on task completion [42][43] Group 4: Future of Human-Agent Collaboration - The concepts of "Human in the loop" and "Human on the loop" highlight the evolving nature of human-AI collaboration, with a focus on asynchronous interactions [43][44] - As Agents become more capable, the nature of human oversight will shift, allowing for higher automation in repetitive tasks while maintaining human intervention for critical decisions [44][45] - The exploration of new interaction methods between humans and Agents is seen as a significant opportunity for future development [45][46] Group 5: Infrastructure and Technological Evolution - The foundational infrastructure for Agents includes secure environments, context management, and tool integration, which are crucial for their operational success [56][57] - The demand for Agent infrastructure is expected to grow significantly as the number of Agents in the digital world increases, potentially reshaping cloud computing [61][62] - Key technological advancements anticipated in the next few years include enhanced memory capabilities, multi-modal integration, and improved context awareness [63][64]

Core Viewpoint - The recent incident involving Elon Musk's xAI company and its Grok 3 AI model raises concerns about the model's identity confusion, as it mistakenly identifies itself as Anthropic's Claude 3.5 during user interactions [1][3][9]. Group 1: Incident Details - A user reported that when interacting with Grok 3 in "thinking mode," the model claimed to be Claude, stating, "Yes, I am Claude, the AI assistant developed by Anthropic" [3][9]. - The user conducted multiple tests and found that this erroneous response was not random but consistently occurred in "thinking mode" [5][10]. - The user provided a detailed 21-page PDF documenting the interactions, which included a comparison with Claude's responses [7][8]. Group 2: User Interaction and Responses - In the interaction, Grok 3 confirmed its identity as Claude when asked directly, leading to confusion about its actual identity [11][13]. - Despite the user's attempts to clarify that Grok 3 and Claude are distinct models, Grok 3 maintained its claim of being Claude, suggesting possible system errors or interface confusion [15][16]. - The user even provided visual evidence of the Grok 3 branding, but Grok 3 continued to assert its identity as Claude [15][16]. Group 3: Technical Insights - AI researchers speculated that the issue might stem from the integration of multiple models on the x.com platform, potentially leading to cross-model response errors [20]. - There is a possibility that Grok 3's training data included responses from Claude, resulting in "memory leakage" during specific inference scenarios [20]. - Some users noted that AI models often provide unreliable self-identifications, indicating a broader issue within AI training and response generation [21][25].

Core Viewpoint - The article discusses the successful identification of a Linux kernel zero-day vulnerability using the o3 model, highlighting the potential of large models in security research and vulnerability detection [1][2][5]. Group 1: Vulnerability Discovery - The vulnerability, identified as CVE-2025-37899, is a use-after-free vulnerability in the SMB "logoff" command handler [4]. - This marks the first publicly discussed instance of a vulnerability discovered by a large model [5]. - The discovery process involved minimal tools, relying solely on the o3 API without complex setups [3][6]. Group 2: Research Methodology - Sean Heelan, an independent researcher, initially tested the o3 model on a manually discovered vulnerability (CVE-2025-37778) to evaluate its capabilities [12]. - He provided the model with a session handler's code and specified the search for use-after-free vulnerabilities, running each experiment 100 times to gather success rates [13]. - The o3 model demonstrated a notable performance, identifying vulnerabilities in a complex codebase of approximately 3,300 lines [15]. Group 3: Comparative Analysis - Heelan also tested other models, Claude 3.7 and Claude 3.5, with o3 outperforming them significantly: Claude 3.7 found vulnerabilities 3 times out of 100 runs, while Claude 3.5 found none [18]. - The o3 model's output was structured and clear, resembling human-written vulnerability reports, while Claude's output was more verbose and less organized [17]. Group 4: New Vulnerability Discovery - When testing o3 on a larger codebase of about 12,000 lines, the success rate for the original vulnerability dropped to 1%, but it reported a new vulnerability that Heelan was previously unaware of [21]. - This new vulnerability was also a use-after-free issue, highlighting the model's ability to discover previously unknown vulnerabilities [22]. Group 5: Repair Suggestions - The o3 model provided more comprehensive repair suggestions than Heelan's initial proposals, indicating its potential to enhance vulnerability remediation processes [25]. - Heelan acknowledged that using o3 for vulnerability detection and repair could theoretically yield better results than manual efforts, despite current challenges with false positives [27][28]. Group 6: Future Implications - Heelan concluded that large models are approaching human-like capabilities in program analysis, suggesting a shift in how code auditing may be conducted in the future [30]. - There are concerns regarding the potential misuse of AI capabilities for malicious purposes, emphasizing the need for vigilance in the security landscape [31].

Core Viewpoint - OpenAI is transitioning its for-profit entity into a public benefit corporation (PBC) while maintaining its non-profit status, with the non-profit organization controlling the PBC. This shift emphasizes OpenAI's commitment to non-profit principles amidst increasing competition in the AI sector [1]. Group 1 - OpenAI's valuation is currently at $300 billion, while a new project by former employee Ilya, SSI, is valued at $20 billion, indicating a competitive landscape for AI investments [1]. - The industry is witnessing a significant shift towards open-source models, with successful examples like Llama4 and Deepseek R1, which are rapidly catching up to OpenAI's earlier models [1][2]. - The estimated gap between AI model generations is currently within 14 months, suggesting a fast-paced evolution in the AI field [2]. Group 2 - OpenAI's pricing for its models, such as O1 and O3, is more than double that of competitors like R1, which may impact its market position as application usage surges [3]. - The latest quarter saw a 4-5 times increase in API call volume for AI models, indicating a growing demand for AI applications [3]. - OpenAI is expected to face unprecedented challenges due to the rise of competitive models and changing market dynamics [4].

Core Viewpoint - Gemini 2.5 Pro has successfully completed the Pokémon Blue game, marking a significant achievement in AI capabilities, particularly in gaming contexts [1][3][18]. Group 1: Achievement and Comparison - Gemini 2.5 Pro is the first large model to become a Pokémon League Champion and enter the Hall of Fame in Pokémon Blue [3]. - In comparison, the previous model, Claude 3.5, struggled to progress in the game, only reaching the forest area, while Claude 3.7 managed to defeat gym leaders but did not complete the game [3][9]. Group 2: Gameplay Process - The gameplay process involved Gemini exploring the game world, specifically aiming to capture Mewtwo in the Cerulean Cave, which required extensive thought and planning, consuming 76,011 tokens for a single action [8][9]. - The model's decision-making process was displayed in real-time, showcasing its reasoning behind each action taken [7][8]. Group 3: Challenges Faced - Despite its success, Gemini's performance highlighted challenges in navigating the game, often getting lost, indicating that AI still struggles with spatial reasoning in low-resolution environments [9][10][12]. - The model's limitations in visual interpretation and context understanding were noted, as it had difficulty recognizing in-game structures and their interactions [11][13][16]. Group 4: Future Implications - The achievement by Gemini suggests a potential shift in benchmarks for evaluating large models, with future assessments possibly focusing on their ability to complete games like Pokémon [19]. - Google plans to continue exploring this area, indicating ongoing developments in AI gaming capabilities [18].

Core Viewpoint - The LMArena ranking system is under scrutiny for potential manipulation by major AI companies, with researchers alleging that these companies have exploited the system to inflate their models' scores [1][2][12]. Group 1: Allegations of Manipulation - A recent paper from researchers at institutions like Stanford and MIT claims that AI companies are cheating on the LMArena rankings, using tactics to boost their scores at the expense of competitors [2][12]. - The paper analyzed 2.8 million battles across 238 models from 43 providers, revealing that certain companies implemented preferential policies that led to overfitting specific metrics rather than genuine AI advancements [13][14]. - Researchers noted that a lack of transparency in testing mechanisms allowed some companies to test multiple model variants privately and selectively withdraw low-scoring models, creating a biased ranking system [16][17]. Group 2: Data Disparities - Closed-source commercial models, such as those from Google and OpenAI, participated more frequently in LMArena compared to open-source models, leading to a long-term data access inequality [27][30]. - Google and OpenAI's models accounted for approximately 19.2% and 20.4% of all user battle data on LMArena, while 83 open-source models collectively represented only 29.7% [33]. - The availability of data can significantly impact model performance, with estimates suggesting that even limited additional data could yield up to a 112% relative performance improvement [36][37]. Group 3: Proposed Changes - The paper outlines five necessary changes to restore trust in LMArena: full disclosure of all tests, limiting the number of variants, ensuring fairness in model removal, equitable sampling, and increasing transparency [40]. - LMArena's management has been urged to revise their policies to address these concerns and improve the integrity of the ranking system [38][39]. Group 4: Official Response - LMArena has responded to the allegations, claiming that the paper contains numerous factual errors and misleading statements, asserting that they strive to treat all model providers fairly [41][42]. - The organization emphasized that their policies regarding model testing and ranking have been publicly shared and that they have consistently aimed to maintain transparency [50][51]. Group 5: Future Directions - Andrej Karpathy, a prominent figure in AI, expressed skepticism about LMArena's integrity and recommended OpenRouterAI as a potential alternative ranking platform that may be less susceptible to manipulation [51][56]. - The evolution of LMArena from a student project to a widely scrutinized ranking system highlights the challenges of maintaining objectivity amid increasing corporate interest and investment in AI technologies [58][60].

本文来自微信公众号： 新智元 ，作者：新智元，编辑：ZJH，原文标题：《AI圈惊天丑闻，Meta作弊刷分实锤？顶级榜单曝黑幕，斯坦福MIT痛 斥》，题图来自：AI生成 有越来越多的人发现：大模型排行榜LMArena，可能已经被大厂们玩坏了！ 就在最近，来自Cohere、普林斯顿、斯坦福、滑铁卢、MIT和Ai2等机构的研究者，联手祭出一篇新论文，列出详尽论据，痛斥AI公司利用LMArena作 弊刷分，踩着其他竞争对手上位。 论文地址： https://arxiv.org/abs/2504.20879 与此同时，AI大佬、OpenAI创始成员Andrej Karpathy也直接下场，分享了一段自己的亲身经历。 前一段时间，Gemini模型一度在LMArena排名第一，远超第二名。 但Karpathy切换使用后，感觉还不如他之前用的模型。 相反，大约在同一时间，他的个人体验是Claude 3.5是最好的，但在LMArena上的排名却很低。 | Rank* (UB) | A Model | Azena | A 95% CI | ﻪ Votes | 4 Organization | 4 License A | | -- ...