You don't know what your agent will do until it's in production. When you ship traditional software to production, you have a pretty good sense of what's going to happen. Users click buttons, fill out forms, move through paths you've already mapped out.When something breaks, you look through stack traces and logs. Agents are different. The input here is natural language, so the range of things people can ask is basically unlimited.The models behind them are sensitive to small changes in phrasing, can give y ...

Welcome to our webinar. We will we will let people trickle in, but today we will be talking about production monitoring for agents. Um so this is the second one that we've done recently or at least that I've done recently.Um the first one was more focused on some of the testing and evals and debugging of agents. And now in this one, we're going to be talking pretty heavily about monitoring agents in production, which we think is a bit of a different uh story and different different kind of like part of of t ...

Introducing Langraph Deploy. Deploy an agent to Langsmith in just a single command. You can go from prototype to production in minutes, all without leaving your terminal.In this video, we're going to walk through that whole workflow from scaffolding a new agent to testing it out locally in Linksmith Studio to finally deploying it using our CLI. Let's go ahead and get right into it. All right, so let's go ahead and get started from scratch.The first thing that we're going to do is run uv tool install langrap ...

Continual Learning in AI Agents - The industry recognizes the gap between AI agents and human learning capabilities, emphasizing the need for agents to continually learn and improve over time [1] - The industry is exploring different methods for AI systems to learn, including weight updates and learning in context using large language models (LLMs) [2] - Reflection over trajectories is emerging as a key theme, allowing agents to update memories, core instructions, and learn new skills [3][4][5] Skill Learning and Implementation - Skill learning involves reflecting over trajectories to learn skills, exemplified by the skill creator skill adapted from Anthropic [8][9] - Deep agent CLI allows specifying environment variables for logging traces, which is useful for reflection [10][11] - The industry is using Langsmith Fetch to grab recent threads from deep agents for reflection and persistent skill creation [12][13] - A practical example demonstrates how an agent can read a JSON file, reflect on its contents, and create a new deep agent skill, showcasing the utility of continual learning [15][16][17] Benefits and Future Directions - Skill learning enables agents to encapsulate standard operating procedures, such as grabbing Langsmith traces, for repeated use [19][20] - Continual learning loop involves agents reflecting on past trajectories to learn facts, memories, skills, and improve instructions [21][22]

Memory Updating Mechanisms for Agents - Explicit memory updating involves directly instructing the agent to remember specific information, similar to how cloud code functions [2][5][6][29] - Implicit memory updating occurs through the agent learning from natural interactions with users, revealing preferences without explicit instructions [7][19][29] Deep Agent CLI and Memory Management - Deep agents have a configuration home directory with an `agent MD` file that stores global memory, similar to Claude's `cloud MD` [3][4][6] - The `agent MD` files are automatically loaded into the system prompt of deep agents, ensuring consistent memory access [6] - Deep agent CLI allows adding information to global memory using natural language commands, updating the `agent MD` file [5] Implicit Memory Updating and Reflection - Agents can reflect on past interactions (sessions or trajectories) to generate higher-level insights and update their memory [8][9][10][28] - Reflection involves summarizing session logs (diaries) and using these summaries to refine and update the agent's memory [11][12] - Accessing session logs is crucial for implicit memory updating; Langsmith can be used to store and manage deep agent traces [13][14][15] Practical Implementation and Workflow - A utility can be used to programmatically access threads and traces from Langsmith projects [21] - The deep agent can be instructed to read interaction threads, identify user preferences, and update global memory accordingly [24][25] - Reflecting on historical threads allows the agent to distill implicit preferences and add them to its global memory, improving future interactions [26][27][28]

Hey, I'm Tanish from Langchain. Today I'm going to go through how to use Open Router's new broadcast feature with Langchain to send traces to Langmith. The cool thing about broadcast is it stores destination information, which in this case is Langmith server side.So the only thing that you need to worry about in your code is your open router API key. Let's go through how to set this up. So let me walk you through a quick code snippet.This uses lang chain's init chat model in order to initialize a model and ...

Evals are an important step for having confidence in the LLM application or agent that you're putting out into the real world is performing as you would expect it to and they're hard to do right. Eugene Yan is one of the people in the business who I think is the sharpest around evals and I saw this tweet from him over the weekend. After repeating myself for the nth time on how to build product evals, I figured I should write it down.It's just three basic steps. One, labeling a small data set. two, aligning ...

AI Workflow Automation & Observability - N8N is an AI workflow builder that allows users to string together nodes into AI agents and set up external triggers for automated execution [1] - Langsmith is an AI observability and evaluation product designed to monitor the performance of AI applications [2] Integration & Setup - Connecting N8N to Langsmith requires generating a Langsmith API key and setting it in the N8N deployment environment [3][8] - Additional environment variables can be set to enable tracing to Langsmith, specify the trace destination, and define the project name [4] Monitoring & Debugging - Langsmith traces provide visibility into the workflow, including requests to OpenAI, model usage, latency, and token consumption [6] - Langsmith offers a monitoring view to track app usage, latency spikes, error rates, and LLM usage/spending [7]

Agent Architecture & Functionality - The Langchain deep research agent is highly configurable and open source, allowing for customization to specific use cases [1] - The agent operates in three main phases: scoping the problem, research, and report writing [3] - The research phase utilizes a supervisor to delegate tasks to sub-agents for in-depth research on specific subtopics [4] - Sub-agents use a tool calling loop, which can be configured with default or custom tools (like MCP servers) for searching flights, hotels, etc [17][18] - A compression step is used by sub-agents to synthesize research findings into comprehensive mini-reports before returning to the supervisor, mitigating context window overload [21][23] - The supervisor analyzes findings from sub-agents to either complete research or continue with follow-up questions [25] - Final report generation is done in a single shot using all condensed research findings [5][27] Implementation & Configuration - The agent is built on Langraph and can be run locally by cloning the Open Deep Research repository [29] - Configuration involves setting API keys for models (default OpenAI) and search tools (default Tavily) [30] - Langraph Studio can be used for iterating and testing the agent with different configurations [32] - The agent is highly configurable, allowing users to choose between default or model provider native search tools, connect to MCP servers, and select models for different steps [33][34] Application & Output - The agent can be used for complex research tasks, such as planning a trip, by iteratively calling tools and searching the web [2] - The agent provides a final report with an overview, flight options, transportation options, accommodation options with booking links, a sample itinerary, and a list of sources [36] - Open Agent Platform provides a UI to configure and try out the research agent without cloning the code [37]

Agentic Platform Engineering Implementation - Outshift by Cisco is redefining platform engineering with agentic platform engineering, aiming to automate workflows and allow platform engineers to focus on high-value work [1][2] - The company built Jarvis, a genai-powered multi-agentic system, to automate tasks and improve efficiency [2] - Langraph, combined with Langsmith's observability tools, enables debugging agentic applications and improving reasoning capabilities at scale [6] Efficiency Improvement - Implementing Jarvis allows developers to self-service through genai-powered automation, eliminating manual toil [4] - CI/CD pipeline setup time reduced from a week to less than an hour [7] - Resource provisioning time (e.g., S3 buckets, EC2 instances, access keys) reduced from half a day to nearly instantaneous [7] - The company has eliminated unnecessary back and forth between developers and SREs [7] Workflow Transformation - The company shifted from traditional automation to agentic reasoning-based workflows by adopting Langraph [5] - Developers interact with Jarvis for platform-related questions and configurations, retrieving information autonomously [8] - The new system allows the company to handle a higher volume of requests with the same team while reducing burnout [8] Technology Evaluation - Langraph's tight integration with Langsmith, especially for debugging and evaluations, is a significant advantage [9] - The company found Langraph to be superior compared to other agentic solutions or custom-built alternatives [9]