AI Model Evaluation (Eval) Industry Trends - Eval remains a core driver for building great AI software, expected to be relevant in the future [2] - The implementation of running evals has changed significantly and will continue to evolve [2] - Updates based on eval results have transitioned from slow and manual to fast and manual [3] - The industry anticipates a shift towards faster updates that are partially or entirely automatic [3] Future of Human-AI Interaction in Evals - Human interaction with evals will evolve from analyzing dashboards to a collaborative process with LLM systems suggesting changes [3][4] - LLM systems may contextualize why changes should be made based on eval results [4] Brain Trust's Perspective - Brain Trust was founded partly due to the lack of significant changes in evals prior to its inception [1] - Brain Trust is excited about the anticipated shift in how humans interact with evals [4]

Build vs Buy Agentic Applications - Whether to build or buy agentic applications depends on what's core to the business [1] - Every agent is quite different and needs to be customizable, especially for different industries like airlines, car rentals, and banks [2] - Providing tools for customization is crucial [2] Challenges in Building Agentic Applications - Building a production-grade agentic application is difficult, even though building a demo is easy [3] - Prioritizing building an agentic application internally may not make sense unless it's an existential threat to the business [3] - The talent required to build production-grade applications is hard to find [3] Internal Development - Non-technical people can build apps, with some becoming power users [3]

Core Concept of Multi-Agent Interaction - Multi-agent interaction is essential for achieving Artificial General Intelligence (AGI) [1] - True multi-agent interaction requires asymmetry between agents [2] Key Elements for Multi-Agent Systems - Knowledge asymmetry, such as robots in different locations gathering different sensor data, is a valuable scenario [3] - Information silos due to privacy concerns or company boundaries necessitate communication protocols for agents to collaborate [3][4] Practical Applications - Scenarios include robots gathering data in different locations [3] - Inter-company interactions where information is guarded due to privacy [3]

Are coding agents really just creating a bigger problem for production. I think the problem in my opinion with VIP coding is that it isn't going far enough, right. Um, and yeah, you know, you've gotten to this place where you can build code quickly.Um, and then you deploy it, something breaks, but now you don't have that deep understanding of like what you actually built, right. And somebody else has to go and fix that and then that learning doesn't sort of transfer back through, right. Um but maybe one way ...

I'm already using codegen tools like cursor. Can I just extend that to solve my production problems. >> Codegen tools are sort of, you know, designed to operate on the sort of the addressible universe of code, right.Production system is sort of like a living breathing animal, right. It's more than just code, right. It's it's really sort of emergent behavior that comes from like a bunch of these things interacting with each other, right. Like the code, the infrastructure, the deployments, the you know the th ...

Agentic System Evolution - The industry views agentic systems as a natural evolution of AI software, which itself is a natural evolution of traditional software [1] - Agentic systems can lead to dramatically simpler code and logic compared to previous generations of software, despite being more powerful and capable of more sophisticated tasks [2] Complexity Assessment - The industry suggests that evaluating agentic systems is not necessarily more complex than evaluating traditional software or single-shot LLM outputs [1] - The resulting agentic systems built by sophisticated customers are often simpler than their predecessors [2]

Future Security Landscape - The industry envisions a behavioral security platform capable of monitoring and protecting all identities, including human, non-human, and AI agents, across an organization [1] - AI agents are expected to become a larger part of the workforce, necessitating protection similar to human employees, both from external threats and from themselves [2] Security Strategy - The company aims to establish an autonomous blue team for continuous exposure detection and mitigation, proactively defending against attackers [2] - The goal is to create a proactive and continuous defense mechanism that balances the power dynamic between attackers and defenders [2]

Core Problem & Solution - Fable Security addresses the critical issue of human error in cybersecurity, which is a leading cause of breaches, especially with the rise of AI-driven attacks [2][3] - Fable Security offers an AI-first human risk platform that understands human vulnerabilities and deploys personalized behavioral interventions to reduce human errors and improve overall organizational security [3][10] - The company shifts the focus from traditional security awareness training to human risk management, providing more relevant and effective protection for employees [5][7] Product & Technology - Fable Security uses AI to understand customer risk at scale and deliver personalized interventions to end-users, addressing specific organizational hot pockets [10] - The platform proactively briefs campaign staffers on the latest breaches and ensures devices are up-to-date, preventing iOS and Android attacks [11] - Fable is constantly monitoring the latest attack techniques, even before security teams have the tools to spot them [5] Customer Impact & Adoption - Fable Security protects enterprises across various sectors, including financial services, healthcare, technology, logistics, manufacturing, and energy [9] - Customers have observed radical results compared to traditional security awareness programs, with employees demonstrating less repeat offending behavior, such as clicking on fewer links and downloading fewer attachments [9][10] - The Democratic National Committee (DNC) used Fable to proactively brief campaign staffers and secure their devices during the election [11] Investment & Growth - Fable Security raised $31 million in its first year [13] - The company is planning to scale the team significantly [14] - Fable offers a unique opportunity for AI developers to apply their knowledge to solve the challenging problem of understanding and influencing human behavior for the better [14]

Core Problem & Solution - Fable Security targets reducing human errors, a leading cause of breaches, with an AI-first human risk platform [2][3] - The platform understands human vulnerabilities and uses AI to deploy personalized behavioral interventions to improve organizational security [3][10] - Fable shifts from security awareness training to human risk management, addressing the ineffectiveness of traditional methods [5][7] AI's Impact on Threat Landscape - Attackers are using AI to create hyper-realistic social engineering attacks, making them harder to detect [3][4] - Fable focuses on identifying and neutralizing these new attack techniques before traditional security measures can [5] Customer Adoption & Results - Fable protects enterprises across various sectors, including financial services, healthcare, technology, logistics, manufacturing, and energy [9] - Customers have observed radical results compared to traditional security awareness programs, with employees committing fewer repeat offenses [9][10] - Examples include proactively briefing employees to prevent ransomware attacks and protecting the Democratic National Committee during elections [11][12] Company Growth & Opportunity - Fable has raised $31 million in its first year [14] - The company is scaling its team and offers an opportunity to apply AI to understand and influence human behavior for the better [15]

Agentic AI 领域 - Greylock 举办 Change Agents 系列讲座，探索 Agentic AI 的前沿技术 [1] - 讲座邀请 Sierra 研究主管兼普林斯顿大学计算机科学副教授 Karthik Narasimhan [1] 多智能体交互 - Karthik Narasimhan 讨论了语言智能体和多智能体交互的未来 [1] - Karthik Narasimhan 探讨了多智能体的定义、能力以及其实验室为推进该领域所做的工作 [1]