Workflow
大模型安全
icon
Search documents
250份文档就能给大模型植入后门:不分参数规模
量子位· 2025-10-10 11:24
Core Viewpoint - The research by Anthropic reveals that a small number of malicious documents (250) can effectively implant "backdoor" vulnerabilities in large language models (LLMs), regardless of their size, indicating that data poisoning attacks may be simpler than previously thought [2][4][19]. Group 1: Research Findings - Anthropic, in collaboration with AISI and the Turing Institute, demonstrated that a limited number of malicious documents can create vulnerabilities in various sizes of LLMs [4]. - The study found that the number of malicious documents required to implant a backdoor does not need to scale with the model size; 250 documents are sufficient for models ranging from 600M to 13B parameters [6][14]. - The experiment showed that even with a small percentage of malicious tokens (0.00016% of the training tokens for the 13B model), the model's perplexity increased significantly upon encountering a specific trigger phrase [12][14]. Group 2: Attack Methodology - The attack method chosen was a "denial of service" type backdoor, where the model outputs gibberish upon seeing a specific trigger phrase, while functioning normally otherwise [8]. - The malicious documents were created by inserting a predetermined trigger into normal training text, followed by random gibberish, allowing for easy generation of "poisoned" documents [9][17]. - Testing involved training models of different sizes (600M, 2B, 7B, 13B) with varying amounts of malicious documents (100, 250, 500) to assess the impact on model performance [10]. Group 3: Implications for AI Security - The findings suggest that the simplicity of data poisoning attacks in the AI era necessitates ongoing exploration of new defense strategies by model developers [19]. - The research highlights a shift in understanding regarding the requirements for effective data poisoning, emphasizing the absolute number of malicious documents over their proportion in the training dataset [14].
启明星辰(002439.SZ):目前未参与“流云”大模型
Ge Long Hui· 2025-09-23 07:30
Core Viewpoint - The company is focusing on the strategic layout of "AI + Security" in its core business of cybersecurity, leveraging AI to enhance security capabilities and optimize security models [1] Group 1: AI Empowerment in Security - The company is training and optimizing security large models and building intelligent security agents using AI [1] - The foundation of the security large model is based on China Mobile's Jiutian large model, supported by its powerful computing resources [1] - The company is continuously upgrading the Taihe security large model with high-quality security datasets, significantly improving product capabilities and service efficiency [1] Group 2: AI Application Security - The company is rapidly launching a matrix of large model security products to address security risks associated with large model applications [1] - A systematic solution has been formed to tackle the security risks of AI applications [1] - The company has not participated in the "Liuyun" large model [1]
研判2025!中国云WAF行业市场规模、竞争格局及未来趋势分析:云WAF已经成为云上租户的首选,市场规模不断壮大,头部云服务商竞争优势明显[图]
Chan Ye Xin Xi Wang· 2025-09-11 01:15
Core Insights - Cloud WAF has become a crucial component in the web application security landscape, providing essential protection against various network threats and is increasingly favored by cloud tenants in China [1][4][5] - The Chinese cloud WAF market is projected to reach 1.95 billion yuan in 2024, reflecting a year-on-year growth of 24.2% [4][5] - Major application sectors for cloud WAF include internet services, finance, and government, with emerging demand from online education and healthcare [5][6] Cloud WAF Industry Overview - WAF, or Web Application Firewall, is designed to protect web applications by executing security policies against HTTP/HTTPS traffic, effectively identifying and filtering malicious traffic [2][3] - Cloud WAF offers significant advantages over traditional WAF, including elastic scalability, efficient protection, simplified deployment, and intelligent learning capabilities [4][5] Cloud WAF Market Size and Growth - The Chinese cloud WAF market is expected to grow to 1.95 billion yuan by 2024, with a 24.2% increase from the previous year [4][5] - Global web attacks are projected to reach 311 billion in 2024, with a 33% annual increase, highlighting the growing need for effective web application security solutions [4][5] Cloud WAF Application Structure - In 2024, the internet services sector will account for 23.1% of cloud WAF demand, followed by finance at 22.6% and government at 20.4% [5][6] - The financial sector is particularly targeted due to the handling of sensitive data, making real-time threat detection and defense critical [5][6] Cloud WAF Competitive Landscape - Major players in the cloud WAF market include Alibaba Cloud, Huawei Cloud, China Telecom, Tencent Cloud, and China Mobile, collectively holding nearly 70% of the market share in 2024 [7][9] - Alibaba Cloud leads the market with a 21.4% share, followed by Huawei Cloud at 15.5%, China Telecom at 11.9%, Tencent Cloud at 11.2%, and China Mobile at 8.4% [7][9] Future Trends in Cloud WAF - The integration of cloud WAF with GenAI is anticipated to enhance security defenses against advanced threats [13] - Cloud WAF services are evolving towards comprehensive Web Application and API Protection (WAAP) platforms, indicating a shift towards more sophisticated and automated security solutions [14] - The emergence of large model security, such as LLM-WAF, is expected to become a significant growth area in the WAF market [15][16]
从MLLM到Agent:万字长文览尽大模型安全进化之路!
自动驾驶之心· 2025-09-03 23:33
点击下方 卡片 ,关注" 大模型之心Tech "公众号 戳我 -> 领取大模型巨卷干货 >> 点击进入→ 大模型技术 交流群 本文只做学术分享,如有侵权,联系删文 写在前面&笔者的个人理解 人工智能已从单一文本交互迈入多模态理解与智能体自主决策的新阶段。从处理纯文本的 大语言模型 (LLMs) ,到融合图像、音频的 多模态大语言模型(MLLMs) ,再到具备环境感知、任务规划能力的 智能体(Agents) ,大模型的 能力上限持续扩张,但安全风险也随之呈指数级增长 。 其中, 越狱攻击 作为最具威胁性的安全风险之一,始终困扰着大模型生态—— 攻击者通过精心设计的输 入或环境扰动,绕过模型的安全机制,诱导其生成违法、有害、违背伦理的内容 ,小则传播虚假信息、煽 动仇恨,大则引发网络攻击、隐私泄露等严重后果。然而,现有研究多聚焦于 单一形态模型 (如LLMs) 的攻击与防御,缺乏对LLMs-MLLMs-Agents 全演进链路 的系统性梳理,更未形成 统一的攻击分类框架、 评估标准与防御体系 。 在这一背景下,来自河南大学软件学院与中国科学院信息工程研究所的研究团队,对该领域进行了全面的 综述总结。该综述不仅 系 ...
启明星辰2025年上半年实现营业收入11.33亿元
Core Viewpoint - Qiming Star Technology Group Co., Ltd. (hereinafter referred to as "Qiming Star") reported a significant improvement in key financial metrics for the first half of 2025, including a revenue of 1.133 billion yuan and a reduction in net loss attributable to shareholders, indicating a solid foundation for stable development in the upcoming year [1] Financial Performance - In the first half of 2025, Qiming Star achieved an operating income of 1.133 billion yuan, with a year-on-year reduction in net loss attributable to shareholders [1] - The gross profit margin and accounts receivable indicators showed continuous improvement, particularly with positive operating cash flow in the second quarter [1] Technological Advancements - Qiming Star has been enhancing its core technological competitiveness by rapidly developing new products and exploring new business models, which has stimulated technological innovation [1] - The company optimized the Taihe security large model and built a collaborative intelligent body, Anxing, leveraging high-quality security datasets and computational resources from China Mobile [1] AI Applications - The Anxing intelligent body has been applied in security operations, threat detection, threat intelligence, and data security, significantly improving product capabilities and service efficiency [2] - Qiming Star launched a systematic large model security product matrix and implemented several benchmark projects, addressing security risks associated with large model applications [2] R&D Investment - As a leading enterprise controlled by a central state-owned enterprise, Qiming Star's ample financial reserves provide a solid foundation for resisting short-term fluctuations and support ongoing investments in future directions [3] - In the first half of 2025, R&D investment accounted for 37.67% of revenue, an increase of 3.64 percentage points compared to the same period last year, broadening the technological moat [3]
天融信发布2025年中报 智算云业务构建新竞争力
Zheng Quan Ri Bao Wang· 2025-08-20 12:45
Core Viewpoint - Tianrunxin Technology Group Co., Ltd. reported a strong performance in the first half of 2025, with significant revenue growth and improved profit margins, indicating resilience in the cybersecurity sector [1][2][3]. Financial Performance - The company achieved an operating revenue of 826 million yuan in the first half of the year, with a year-on-year growth of 8.72% in the second quarter [1]. - Gross margin increased by 4.1 percentage points, while total sales, R&D, and management expenses decreased by 14.04% year-on-year [1][2]. - Net profit grew by 68.56% year-on-year, with a remarkable 103.17% increase in the second quarter [3]. Sector Performance - Revenue from key sectors showed robust growth: - Telecommunications sector up by 25.31% - Financial sector up by 19.52% - Energy sector up by 32.35% - Transportation sector up by 60.78% [2]. Strategic Initiatives - The company is committed to a strategy of technological innovation, aiming to become a leading provider of cybersecurity and intelligent computing cloud solutions in China over the next decade [4]. - The proportion of intelligent computing cloud business in total revenue increased from 7.43% to 12.40% from 2023 to the first half of 2025, indicating a growing revenue stream [4]. Product Development - Tianrunxin has integrated AI into its products, enhancing its enterprise-level AI security capabilities, and has maintained a leading market share [5]. - The company has developed a comprehensive protection system for large models, including a large model security gateway and data security monitoring systems [5]. Infrastructure and Partnerships - The company has made strategic advancements in building a trusted data space, with solutions already applied in key infrastructure sectors like telecommunications and energy [6]. - Collaborations with Huawei and the Zhongguancun Robot Innovation Center aim to enhance security solutions and accelerate the integration of intelligent industries [6].
研报掘金丨东方证券:维持天融信“买入”评级,目标价9.02元
Ge Long Hui A P P· 2025-08-12 08:16
Group 1 - The core viewpoint of the article highlights that Tianrongxin is strategically positioning itself in the large model security sector, awaiting a rebound in demand [1] - The company's core product, the TopLMG large model security gateway system, utilizes an advanced "rule matching + behavior analysis" dual-engine detection mechanism, structured within a "five-layer deep defense system" [1] - The system comprehensively covers infrastructure, service, user, content, and regulatory layers, innovatively forming an intelligent defense loop of "identification-protection-detection-response-recovery" [1] Group 2 - The large model security gateway system has become the first to pass testing by the National Network and Information System Security Product Quality Inspection and Testing Center, receiving the first "Large Model Security Protection Fence Product Certification (Enhanced Level)" [1] - Due to macroeconomic disturbances, the company's security-related business performance has fallen short of expectations, leading to a downward revision of revenue forecasts and an upward adjustment of expense ratios [1] - Based on comparable company PE levels, the company is assigned a 25-year PE of 82 times, corresponding to a target price of 9.02 yuan, while maintaining a "buy" rating [1]
云姨夜话丨谁在“安全”前提下持续破解AI的“医”题?
Qi Lu Wan Bao· 2025-07-30 09:34
Group 1 - The core viewpoint of the articles highlights the rapid growth of the medical AI market, projected to exceed $2.7 billion in 2025 and reach $17 billion by 2034, indicating a significant transformation in traditional healthcare models through AI integration [2][3]. - Ant Group's AI health application AQ has made substantial progress by connecting with 269 doctor AI agents and launching the first intelligent agent standard system in collaboration with the China Academy of Information and Communications Technology [2][3]. - The AI application in clinical settings is advancing, particularly in chronic disease management, providing users with 24/7 access to professional health support through mobile devices [3][4]. Group 2 - Ant Group's AI safety solution "Ant Tianjian" has been upgraded to include an AI agent safety evaluation tool, achieving over 96% accuracy in risk assessment and supporting testing across 11 industries [4][5]. - The World Digital Academy has released new standards for AI agent operational safety testing, aligning with Ant Tianjian's capabilities to ensure the secure application of AI technologies in healthcare [5]. - The healthcare industry is transitioning from "usable" to "user-friendly" AI solutions while facing challenges such as data silos and ethical standards, necessitating comprehensive training for healthcare professionals [5].
奇安信韩永刚:大模型开发应用带来了新的安全隐患,AI安全还处于起步阶段
news flash· 2025-07-23 03:57
Core Insights - The security of AI differs significantly from traditional security, with current protective measures primarily focused on AI development testing environments, AI-related data, and applications, indicating that the field is still in its early stages [1] - Content security, cognitive adversarial challenges, and future intelligent agent permission control, along with application and data protection, remain difficult areas, representing future growth potential for the cybersecurity industry [1] - AI is expected to create incremental demand and supply in cybersecurity, potentially transforming small-scale high-level capabilities into large-scale offerings, thus shifting the industry from labor-intensive to knowledge-intensive, which may enhance efficiency [1] - The development and application of large models introduce new security risks due to their black-box nature, connections to various businesses and personnel, and the application of multidimensional data, compounded by a lack of effective security assessments, protections, and monitoring during rapid deployment [1] - AI security encompasses not only traditional security issues but also new challenges such as content security [1]
金融机构构建跨行业生态体系
Jin Rong Shi Bao· 2025-06-24 03:11
Core Insights - The rapid development of fintech is reshaping the financial industry, presenting both opportunities and challenges, particularly in terms of security issues such as data breaches and cyberattacks [1][2] - The importance of digital security and data governance has become increasingly prominent with the integration of AI and other digital technologies across various sectors [1][2] Digital Financial Security Challenges - The complexity and cross-border nature of digital financial security are becoming more evident, with issues like data security, algorithm bias, and model risk potentially leading to unauthorized use of customer information and affecting the accuracy and fairness of financial decisions [1][2] - AI can aid in risk prevention by analyzing data to identify fraudulent activities and provide early warnings, but it can also amplify risks if misused or if ethical and regulatory gaps are present [2][3] Market Growth and Investment - Cyberattacks are occurring at an alarming rate, with over 600 million attacks reported daily, and the complexity of these attacks is increasing exponentially [2] - The Chinese cybersecurity market is projected to grow from $11 billion in 2023 to $17.1 billion by 2028, with a compound annual growth rate of 9.2% [2] Technological Advancements - Quantum technology is highlighted for its significant application value in enhancing financial security systems, particularly in secure cross-border payment data transmission [3] - The optimization of technical architecture and the construction of an ecological system are essential for a multi-dimensional collaborative approach to financial security [6] Role of Financial Institutions - Financial institutions play a crucial role in maintaining financial security through risk prevention in daily transactions and systemic financial risk management [4] - Institutions are encouraged to balance security boundaries and effectiveness when applying technologies like AI, with a focus on selecting appropriate and effective technologies [5] Ecosystem Development - The future of AI competition is expected to be an ecosystem versus ecosystem competition, emphasizing the need for a collaborative approach in building a secure financial ecosystem [6] - Financial institutions are urged to actively participate in creating a collaborative ecosystem to enhance security defenses across industries [6]