Hi everyone. Uh just going to give it a few minutes to let everyone kind of filter in and we'll get started. All right.Um, so we'll go ahead and get started. Um, today we're going to be talking primarily about how to, um, essentially debug, evaluate, and ship reliable AI agents. Um and we're going to talk about it in the context of Langmith. Um so a lot of what we see today um is built into our Langmith product in terms of the way to ship reliable and production agents.So I'm excited to jump into all of tho ...

RT Breakout (@breakoutprop)"Pay a fee to trade?" Sounds like gambling.It's not.A surgeon doesn't walk into an OR with their own scalpel on day one.A pilot doesn't buy a 747 to prove they can fly.They go through an evaluation. Prove competence. Then the institution backs them with the tools.A prop eval works the same way.Trade a simulated account. Hit a profit target. Stay within the drawdown limit. Pass.The firm hands you the capital. You keep the majority of profits.The fee isn't a gamble. It's a credentia ...

Anyone can build an agent. But is yours actually working? LangSmith is the agent engineering platform built for observability, evaluation, and everything in between — so you can stop guessing and start shipping better agents. Whether you're building on LangGraph, OpenAI SDK, Anthropic SDK, CrewAI, or anything else — LangSmith connects via OpenTelemetry. No stack changes required. Free to get started → https://smith.langchain.com Learn more about our products → https://langchain.com ...

Two of the most crucial things when building production agents is setting up proper observability and setting up proper evaluation. And these are actually tied and coupled, and this is different than in software engineering and the role observability and evaluation play when building agents is different than in software engineering as well. So I wanna talk a little bit about how we view observability and how it powers a lot of agent evaluation.Maybe starting briefly highlighting some of the things that we t ...

Core Insights - The AI industry in 2025 is focusing on the core narrative of model capabilities, particularly in supporting agents, coding abilities, and effective tool usage, moving beyond mere leaderboard scores to real-world task performance as a new standard for evaluation [2][10] - ByteDance's newly released Doubao model 1.8 enhances agent support capabilities, including coding and tool usage, and introduces an imaginative scenario with OS Agent [4][11] - The introduction of visual capabilities in agents allows them to understand and interact with the world, which is crucial for assisting with complex real-world tasks [8][16] Model Development - Current model advancements are not limited to text-based models; they now include enhanced visual capabilities, allowing models to "see" and comprehend the world [7][10] - Doubao 1.8 combines LLM and VLM capabilities from the outset, achieving significant improvements in visual understanding and maintaining reasoning performance [8][10] - The Doubao model's ability to catch up with the Gemini series in a short time indicates a consensus among foundational model companies regarding the future development of models [10] Agent Capabilities - The emergence of OS Agent has sparked a wave of entrepreneurship in AI agents, with a focus on the reliability of tool invocation becoming a key concern for developers [11][12] - Doubao 1.8 significantly enhances the ability of agents to use tools, which is a common focus among recently released models [12][13] - The core capability of Doubao 1.8 is its OS Agent, which allows it to "see" and interact directly with interfaces, unlocking new use cases [14][16] Evaluation Systems - The evaluation of models is shifting from traditional benchmarks to real-world applications, emphasizing user experience and the ability to perform complex tasks that reflect actual user needs [29][32] - Doubao 1.8's evaluation system prioritizes real-world scenarios and aims to advance general intelligence while ensuring practical usability [35][36] - The challenges of customer service scenarios highlight the complexity of real-world tasks, which require high accuracy and emotional intelligence, showcasing the potential for AI to enhance user experiences [36][40]

Educational Resources - Stanford's CS336 provides a video guide to Karpathy's nanochat, covering essential topics for Frontier AI Labs preparation [1] Key AI Concepts - The curriculum includes Tokenization, Resource Accounting, Pretraining, Finetuning (SFT/RLHF), Key Architectures, GPUs, Kernels, Tritons, Parallelism, Scaling Laws, Inference, Evaluation, and Alignment [1]

Educational Resources - Stanford's CS336 video guide covers topics essential for Frontier AI Labs jobs [1] - The curriculum includes tokenization, resource accounting, pretraining, and finetuning (SFT/RLHF) [1] - Key AI architectures, GPU usage, kernels, parallelism, and scaling laws are addressed [1] AI Development Lifecycle - The guide also covers inference, evaluation, and alignment in AI models [1]

Investment Evaluation - Financial statements possess 12 characteristics crucial for evaluating companies before investing [1] - These characteristics can increase the chances of choosing a winner [1] Resource - A resource is available to discover these 12 characteristics [1]

AI Evaluation Challenges - Traditional evaluation methods are inadequate for assessing GenAI applications' brittleness [1] - The industry faces a "Last Mile Problem" in AI, ensuring reliability, quality, and alignment for any application [1] - Standard evaluation methods often fail to uncover corner cases and unexpected user inputs [1] Haize Labs' Approach - Haize Labs simulates the "last mile" by bombarding AI with unexpected user inputs to uncover corner cases at scale [1] - Haize Labs focuses on Quality Metric (defining criteria for good/bad responses and automating judgment) and Stimuli Generation (creating diverse data to discover bugs) [1] - Haize Labs uses agents as judges to scale evaluation, considering factors like accuracy vs latency [1] - Haize Labs employs RL-tuned judges to further scale evaluation processes [1] - Haize Labs utilizes simulation as a form of prompt optimization [1] Case Studies - Haize Labs has worked with a major European bank's AI app [1] - Haize Labs has worked with a F500 bank's voice agents [1] - Haize Labs scales voice agent evaluations [1]

Product & Technology - Brain Trust introduces "Loop," an agent integrated into its platform designed to automate and improve prompts, datasets, and scorers for AI model evaluation [4][5][7] - Loop leverages advancements in frontier models, particularly noting Claude 4's significant improvement (6x better) in prompt engineering capabilities compared to previous models [6] - Loop allows users to compare suggested edits to data and prompts side-by-side within the UI, maintaining data visibility [9][10] - Loop supports various models, including OpenAI, Gemini, and custom LLMs [9] User Engagement & Adoption - The average organization using Brain Trust runs approximately 13 evaluations (EVELs) per day [3] - Some advanced customers are running over 3,000 evaluations daily and spending more than two hours per day using the product [3] - Brain Trust encourages users to try Loop and provide feedback [12] Future Vision - Brain Trust anticipates a revolution in AI model evaluation, driven by advancements in frontier models [11] - The company is focused on incorporating these advancements into its platform [11] Hiring - Brain Trust is actively hiring for UI, AI, and infrastructure roles [12]