Core Idea - Factory believes software development is transitioning to agent-driven from human-driven [1] - To achieve significant productivity gains (5-20x), a shift from collaborating with AI to delegating tasks entirely to AI is needed [3] - Factory is building a platform for managing and scaling AI agents, integrating various engineering systems [3][4][5] Agentic System Characteristics - Agentic systems require planning to decide future actions [11] - Decision-making is crucial for agents to make calls based on the existing state [13][14] - Environmental grounding is necessary for agents to interact with and adapt to the external environment [14] Human-AI Collaboration - Humans will remain in software development, focusing on the outer loop (reasoning, requirements) [15][16] - Agents will handle the inner loop (coding, testing, code review) [17] - AI UX should blend delegation with control for situations where agents cannot complete tasks [17] Agent Reliability - Clear planning and boundaries are essential for reliable agents [32] - Subtask decomposition, model predictive control, and explicit plan templating can improve planning [19][20] - Control over the tools agents use is the most important differentiator in agent reliability [28] Environmental Interaction - New AI computer interfaces are needed for agents to interact with the world [28] - Processing information from the environment is crucial for complex systems [29][30] - Agents need to ground themselves in the environment to perform full software development work [32] Call to Action - Factory encourages teams not delegating at least 50% of engineering tasks to AI agents to engage with them [34]

LangChain Agent Evaluation - LangChain 降低了 Agent 评估的门槛，使得非开发者也能轻松进行 [1] - Langraph Studio 新增了快速评估 Langraph Agent 的功能 [3] - 用户可以在 Langraph Studio 中选择数据集并启动评估实验 [3][4] - 评估结果可在 Langsmith 中查看，包括模型输出和评估分数 [5] Evaluation Importance and Accessibility - 评估对于构建有效的 Agent 至关重要 [7] - 传统评估对开发者有较高要求，需要掌握 SDK、Piest 和 Evaluate API 等 [7] - LangChain 旨在提供一种无需代码的方式，让任何人都能评估 Langraph Agent [8] - 非技术用户可以基于直觉评估模型选择和提示词等 [9] Configuration and Customization - 用户可以在 Studio 界面中轻松切换 graph 配置，并以此为基础启动评估 [9] - 开发者可以预先设置包含输入主题和参考输出的数据集 [10] - 可以将评估器（Evaluator）绑定到数据集，并自定义评估标准和评分规则 [11][12][13] - 用户可以在 Studio 中修改 graph 配置（如模型、提示词），并启动新的评估实验 [15][16][17] - Studio 提供了无代码配置方式，方便快速迭代 [18]

Company Overview - Vizian serves 97% of academic medical centers in the US, over 69% of acute care hospitals, and more than 35% of the ambulatory market [1] - Vizian is developing a generative AI platform to improve healthcare providers' data access and analysis [2] Challenges Before Langraph and Langsmith - Scaling LLM queries using Azure OpenAI faced token limit issues, impacting performance [3] - Limited visibility into system performance made it difficult to track token usage, prompt efficiency, and reliability [3] - Continuous testing was not feasible, leading to reactive problem-solving [4] - Multi-agent architecture introduced complexity, requiring better orchestration [4] - Lack of observability tools early on resulted in technical debt [4] Impact of Integrating Langraph and Langsmith - Gained the ability to accurately estimate token usage, enabling proper capacity provisioning in Azure OpenAI [5] - Real-time insights into system performance facilitated faster issue diagnosis and resolution [6] - Langraph provided structure and orchestration for multi-agent workflows [6] - Resolved LLM rate limiting issues by optimizing token usage and throughput allocation [7] - Development and debugging processes became significantly faster [8] - Shift to automated continuous testing dramatically improved system quality and reliability [8] - Rapidly turn beta user feedback into actionable improvements [8] Recommendations - Start with a slim proof of concept and model one high impact user flow in Langraph [9] - Integrate with Langsmith from day one and treat every run as a data point [9] - Define a handful of golden query response pairs upfront and use them for acceptance testing [9] - Budget a short weekly review of Langsmith's run history [9]

[Music] I'm Isis Julian and I'm a senior software engineer at Morning Star and I work on the intelligence engine. Morning Star is a global leader in providing investment research data and analysis and we pride ourselves in empowering investor success by serving transparent, accessible, and reliable investment information. So, with AI gaining increasing popularity nearing the end of 2022 and early 2023, with a scrappy team of just five engineers, we were able to launch our first ever AI research assistant na ...

Hey everyone, my name is Shrea. I am finishing up my PhD at UC Berkeley, so that's quite exciting for me. Um, and I'm here to give you a different kind of talk.This is about research, what we're learning through research and how to help people build reliable LLM pipelines. Just to give a picture of the kind of research that we do at Berkeley. This is around data processing agents.What do I mean by data processing. Organizations have lots of unstructured data documents that they want to extract and analyze, ...

Company Overview - 11X is building digital workers, including Alice, an AI SDR, and Julian, an AI voice agent [1] - The company relocated from London to San Francisco and rebuilt its core product, Alice, from the ground up [2] Alice Rebuild & Vision - The rebuild of Alice was driven by the belief that agents are the future [3] - The new vision for Alice centers on seven agentic capabilities, including chat-based interaction, knowledge base training via document uploads, AI-driven lead sourcing, deep lead research, personalized emails, automatic handling of inbound messages, and self-learning [11][12][13] Development Process & Tech Stack - The rebuild of Alice 2 took only 3 months from the first commit to migrating the last business customer [3][14] - The company chose a vanilla tech stack and leveraged vendors like Langchain to move quickly [15][16][17] - Langchain was chosen as a key partner due to its AI dev tools, agent framework, cloud hosting, observability, Typescript support, and customer support [18][19] Agent Architecture Evolution - The company experimented with three different architectures for campaign creation: React, workflow, and multi-agent systems [21] - The final architecture was a multi-agent system with a supervisor and specialized sub-agents for research, positioning, LinkedIn messaging, and email writing [44][45][46] Results & Future Plans - Alice 2 went live in January and has sourced close to 2 million leads and sent close to 3 million messages [52] - Alice 2 has generated about 21,000 replies, with a reply rate of around 2%, on par with a human SDR [52] - Future plans include integrating Alice and Julian, implementing self-learning, and exploring new technologies like computer use, memory, and reinforcement learning [53][54]

Agent Adoption & Scalability - LinkedIn aims to scale agentic adoption within the organization to enable broader idea generation [2] - LinkedIn built the Hiring Assistant, its first production agent, to automate recruiter tasks and free up time for candidate interaction [3] - The Hiring Assistant follows an ambient agent pattern, operating in the background and notifying recruiters upon completion [4][5] - LinkedIn adopted a supervisor multi-agent architecture, with a supervisor agent coordinating sub-agents that interact with LinkedIn services [6] Technology Stack & Framework - LinkedIn standardized on Python for GenAI development, moving away from its traditional Java-centric approach [7][8] - The company built a service framework using Python, gRPC, Langchain, and Langraph to streamline the creation of production-ready Python services [9][19] - Over 20 teams have used this framework to create over 30 services supporting Generative AI product experiences [9][10] - Langchain and Langraph were chosen for their ease of use and sensible interfaces, enabling rapid development and integration with internal infrastructure [22][23] Infrastructure & Architecture - LinkedIn invested in a distributed architecture to support agentic communication modes [10] - The company modeled long-running asynchronous flows as a messaging problem, leveraging its existing messaging service for agent-to-agent and user-to-agent communication [26][27] - LinkedIn developed agentic memory with scoped and layered memory types (working, long-term, collective) [29][30] - LinkedIn implemented a centralized skill registry, allowing agents to discover and access skills developed by different teams [34][35]

Company Overview & Strategy - BlackRock manages over $11 trillion in assets, aiming to enhance financial well-being through its Aladdin platform [3] - Aladdin is a proprietary technology platform unifying the investment management process for both public and private markets, serving institutional and retail investors [3] - BlackRock aims to increase productivity, drive alpha generation, and personalize user experience through AI, particularly with the Aladdin Copilot initiative [5][6] Aladdin Copilot & Architecture - Aladdin Copilot is embedded across 100 front-end applications, proactively surfacing relevant content and enhancing productivity [6][7] - The architecture supporting Aladdin Copilot uses a plug-in registry, allowing 50-60 Aladdin engineering teams to integrate their functionalities [9][10] - The system uses Langchain for orchestration, incorporating input and output guardrails for responsible AI moderation, including off-topic, toxic content, and PII handling [16][24] - GPT4 function calling is used for planning and action within the orchestration node, iterating until a final answer is obtained [19] Evaluation & Testing - BlackRock emphasizes evaluation-driven development, similar to test-driven development in traditional coding, to ensure the system's reliability [25] - The company tests every system prompt and intended behavior by generating synthetic data and building evaluation pipelines [26][27] - End-to-end testing capabilities are provided to developers, allowing them to configure testing scenarios, including application context, user settings, and multi-turn scenarios [30][31] - Ground truth data is collected for every plug-in to guarantee Aladdin Copilot's performance and routing in various scenarios [33]

[Music] I'm Paul Rasgitis and I'm the tech lead for AI products at Modern Treasury. Modern Treasury is the payment operations platform built for the instant economy. At Modern Treasury, our goal is to transform how teams track and move money.Recently, we launched Modern Treasury AI, a horizontal platform that includes chat, task management, monitoring, and an AI agent designed specifically for financial operations. The agent is just one part of this broader experience, but it plays a key role in how users i ...

Agentic Platform Engineering Implementation - Outshift by Cisco is redefining platform engineering with agentic platform engineering, aiming to automate workflows and allow platform engineers to focus on high-value work [1][2] - The company built Jarvis, a genai-powered multi-agentic system, to automate tasks and improve efficiency [2] - Langraph, combined with Langsmith's observability tools, enables debugging agentic applications and improving reasoning capabilities at scale [6] Efficiency Improvement - Implementing Jarvis allows developers to self-service through genai-powered automation, eliminating manual toil [4] - CI/CD pipeline setup time reduced from a week to less than an hour [7] - Resource provisioning time (e.g., S3 buckets, EC2 instances, access keys) reduced from half a day to nearly instantaneous [7] - The company has eliminated unnecessary back and forth between developers and SREs [7] Workflow Transformation - The company shifted from traditional automation to agentic reasoning-based workflows by adopting Langraph [5] - Developers interact with Jarvis for platform-related questions and configurations, retrieving information autonomously [8] - The new system allows the company to handle a higher volume of requests with the same team while reducing burnout [8] Technology Evaluation - Langraph's tight integration with Langsmith, especially for debugging and evaluations, is a significant advantage [9] - The company found Langraph to be superior compared to other agentic solutions or custom-built alternatives [9]