Overview - Generative AI is included in corporate roadmaps, but companies should not release any unsafe products. The threat model has changed due to LLMs, where untrusted natural language can become an attack surface, and outputs can be weaponized. Models should operate in a sandboxed, monitored, and strictly authorized environment [1][2] Security Challenges - Immediate injection attacks, including indirect attacks hidden in files and web pages, are now a top risk for LLMs. Attackers can compromise inputs without breaching backend systems, leading to data theft or unsafe operations [4][5] - Abuse of agents/tools and "over-proxying" create new permission boundaries. Overly permissive agents can be lured into executing powerful operations, necessitating strict RBAC and human approval for sensitive actions [4][5] - RAG (Retrieval-Augmented Generation) introduces new attack surfaces, where poisoned indexes can lead to adversarial outputs. Defensive measures are still evolving [4][5] - Privacy leaks and IP spillage are active research areas, with large models sometimes memorizing sensitive training data. Improvements in vendor settings are ongoing [4][5] - The AI supply chain is vulnerable, with risks from backdoored models and deceptive alignments. Organizations need robust provenance and behavior review measures [4][5] - Unsafe output handling can lead to various security issues, including XSS and SSRF attacks. Strict output validation and execution policies are essential [4][5] - DoS attacks and cost abuse can arise from malicious workloads, necessitating rate limits and alert systems [4][5] - Observability and compliance challenges exist, requiring structured logging and change control while adhering to privacy laws [4][5] - Governance drift and model/version risks arise from frequent updates, emphasizing the need for continuous security testing and version control [4][5] - Content authenticity and downstream misuse remain concerns, with organizations encouraged to track output provenance [4][5] Action Plan for Next 90 Days - Conduct a GenAI security and privacy audit to identify sensitive data entry points and deploy immediate controls [6][7] - Pilot high-value, low-risk use cases to demonstrate value while minimizing customer risk [6][7] - Implement evaluation tools with human review and key metrics before widespread deployment [6][7] Case Studies - JPMorgan Chase implemented strict prompts and a code snippet checker to prevent sensitive data leaks in their AI coding assistant, resulting in zero code leak incidents by 2024 [16] - Microsoft enhanced Bing Chat's security by limiting session lengths and improving prompt isolation, significantly reducing successful prompt injection attempts [17] - Syntegra utilized differential privacy in their medical AI to prevent the model from recalling sensitive patient data, ensuring compliance with HIPAA [18] - Waymo employed a model registry to ensure the security of their machine learning supply chain, successfully avoiding security issues over 18 months [19][20] 30-60-90 Day Action Plan - The first 30 days should focus on threat modeling workshops and implementing basic input/output filtering [22][23] - The next 31-60 days should involve red team simulations and the deployment of advanced controls based on early findings [24][25] - The final phase (61-90 days) should include external audits and optimization of monitoring metrics to ensure ongoing compliance and security [27][28]