AI智能体安全
Search documents
AgentDoG:为AI智能体戴上「诊断项圈」
机器之心· 2026-02-06 03:57
随着 AI 智能体(Agent)能力日益强大,其自主行为带来的安全风险也愈发复杂。现有安全工具往往只能给出「安全 / 不安全」的简单判断,无法告知我们风险的 根源。为此,上海人工智能实验室正式开源 AgentDoG (Agent Diagnostic Guardrail),一个专为 AI 智能体设计的 诊断式安全护栏框架 。它不仅能精准判断 Agent 行 为的安全性,更能 诊断风 险来源、追溯失效 模式、解释决策动因 ,为 AI 智能体的安全发展保驾护航。 当 AI 智能体「放飞自我」,如何确保安全? AI 智能体(Agent)正在从实验室走向现实,它们能自主规划、调用工具、与环境交互,在科研、金融、软件工程等领域展现出巨大潜力。然而,这枚硬币的另一 面是前所未有的安全挑战。 一个能够操作文件、调用 API、访问网络的 Agent,其行为风险不再仅仅是「说错话」。它可能因为一条隐藏在网页中的恶意指令而泄露你的隐私文件,可能因错 误理解工具的参数而造成经济损失,甚至可能在多步操作中「悄无声息」地偏离正轨,执行危险动作。 面对这些 「智能体式」的风险 (Agentic Risks),现有的 guard mode ...
速递|Runlayer已签约数十家客户:三度创业者为Agent系上“安全带”,获1100万美元种子轮融资
Z Potentials· 2025-11-18 02:51
Core Insights - Runlayer, a new AI security startup, has raised $11 million in seed funding from Khosla Ventures and Felicis, and has signed contracts with eight unicorn companies since its product launch four months ago [1][3]. Company Overview - The founder of Runlayer, Andrew Berman, is a serial entrepreneur with previous ventures including Nanit and Vowel, the latter of which was sold to Zapier in 2024 [2]. - Runlayer has attracted notable clients, including Gusto, dbt Labs, Instacart, and Opendoor, highlighting its market traction [3]. Product and Technology - Runlayer's product is based on the Model Context Protocol (MCP), which allows AI agents to autonomously access, migrate, modify data, and execute business processes without human supervision [5]. - The MCP has gained support from major model manufacturers like OpenAI, Microsoft, AWS, and Google, as well as thousands of tech and enterprise clients across various industries [5]. Security Challenges - Despite the growing interest in AI, the MCP framework has been found to have security vulnerabilities, as evidenced by incidents involving GitHub and Asana where sensitive data was exposed [6]. - The emergence of these security issues has led to a surge in demand for MCP security products, with established companies and startups developing solutions to address these vulnerabilities [6]. Competitive Landscape - The most common MCP security products are gateways that authenticate agents and control their access to applications [7]. - Runlayer aims to differentiate itself in a competitive market by offering an integrated security tool that combines various functionalities, including threat detection and activity monitoring [7][8]. Team and Expertise - Berman's experience as the head of Zapier's AI department and his collaboration with OpenAI and Anthropic on early MCP servers provide Runlayer with a unique advantage in understanding and addressing security risks [8][9]. - The company has also secured the involvement of key advisors and investors, including David Soria Parra, the chief creator of the MCP protocol [4][10].
新型数字安全须用AI技术抵御AI攻击
Nan Fang Du Shi Bao· 2025-09-15 23:10
Core Insights - The event focused on the integration of AI and cybersecurity, emphasizing the importance of security governance in the era of artificial intelligence [2][5][7] - The "2025 Guangzhou Cybersecurity Awareness Week" was officially launched, highlighting the city's commitment to enhancing cybersecurity measures in light of upcoming major events [2][9] Group 1: Event Overview - The "2025 Guangzhou Cybersecurity Awareness Week" was successfully launched on September 15, coinciding with the 12th National Cybersecurity Awareness Week [2] - The event was co-hosted by multiple government departments and organizations, aiming to promote high-quality development in cybersecurity [2][3] Group 2: Cybersecurity Initiatives - Guangzhou is accelerating the construction of a modern industrial system that emphasizes technological innovation and cybersecurity [3] - The city aims to strengthen the cybersecurity ecosystem by fostering collaboration among government, industry, academia, and users [3][4] Group 3: AI and Cybersecurity - Experts discussed the challenges posed by AI, including issues of overreach and excessive delegation, and the need for a collaborative approach to ensure AI's safe development [5][6] - The "AI Intelligent Agent Security Development Initiative" was jointly launched by various stakeholders to promote safe and reliable AI technologies [6] Group 4: Upcoming Competitions - The "2025 Yangcheng Cup" cybersecurity competition has opened registration, targeting a diverse group of participants from various educational and professional backgrounds [9][11] - The competition will consist of an online preliminary round and an offline final, designed to assess participants' cybersecurity skills in real-world scenarios [10][12]