An introduction to the evals track About Manu Goyal Manu Goyal is the founding engineer at Braintrust. Previously, he developed autonomous systems at Nuro. He has an 8 year old Pomeranian named Hendrix. Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter ...

[Music] Welcome everyone. Thank you so much for coming. My name is Forest.This is Matt. Uh and we're going to be talking to you today about augment agent and specifically legacy code. how we get the most out of gnarly legacy code bases using an AI agent.So I do not work for Augment Code. Um I am a friend and partner of Augment Code. So I helped to put this talk together.Matt is from Augment Code. So he's going to be your best person to come to with your most detailed technical questions after the session. M ...

[Music] Um, yeah. So, my name is Mike Chambers. I'm going to pick you up a little bit there. So, I'm from Queensland in the eastern part of Australia, but that's okay. Um, yeah, very happy to be here. So, I'm a developer advocate for Amazon Web Services. Um, and I completely and utterly and only and totally spe specialize in generative AI. Used to be machine learning. Now it's generative AI. Um, I'll be talking about why this slide is up here in a moment. Any tabletop RPG players in the room? There's got to ...

As organizations seek to harness their proprietary data while maintaining security and compliance, Amazon Bedrock provides a comprehensive framework for building tailored AI applications. Using Amazon Bedrock Knowledge Bases and Amazon Bedrock Data Automation, organizations can create AI solutions that truly understand their unique business context, terminology, and requirements. Combined with Amazon Bedrock Guardrails, these capabilities enhance the accuracy and relevance of AI-generated responses, while e ...

The Gemini Live API GA is now powered by Google's best cost-effective thinking model Gemini 2.5 Flash. We will do a deep dive on the capabilities that the Gemini Live API combined with Pipecat unlock for devs with special focus on session management, turn detection, tool use (including async function calls), proactivity, multilinguality and integration with telephony and other infra. We will demo some of the more innovative capabilities. We will also talk through some customer use cases - especially how cus ...

[Music] We're here to talk about real time conversational video uh with Pipecat. That's me, and with Tavis, that's Brian. We'll introduce ourselves a little bit more, but in the interest of keeping it moving, let's talk about what we're here for.If anybody Have any of you ever seen one of these robot concierge things. Do they work. No, they don't.They're terrible, right. Um, it's actually possible nowadays to build this kind of thing, but actually good. Um, it's a little bit tricky, but that's what we're he ...

Every vector database out there is both faster and slower than any other competitor — if you believe all the benchmarketing out there. Let's turn the marketing into useful benchmarks that actually help you: 1. How not to benchmark (spoiler: don’t trust the glossy charts). 2. What’s uniquely tricky about benchmarking vector search. 3. How to build meaningful benchmarks tailored to your use case. PS: Yes, you will have to get your hands dirty. Never believe a benchmark that you haven't tweaked yourself. About ...

[Music] So I'm here to talk about taming rogue AI agents but essentially want to talk about uh evaluation driven development observability driven but really why we need observability. So, who uses AI? Is that Jim's stupid most stupid question of the day? Probably. Who trusts AI? Right. If you'd like to meet me after, I've got some snake oil you might be interested in buying. Yeah, we do not trust AI in the slightest. Now, different question. Who reads books? That's reading books. If you want some recommenda ...

Security is the biggest blocker for agent orchestration adoption in regulated industries for SWE agents. Gitpod's agent orchestration went from an originally self-hosted kubernetes architecture to the current 'bring your own cloud' model that enables deployment our SWE agent orchestration platform in secure environments. The architecture allows customers to securely connect their foundational models and agent memory solutions and comes with features like auto-suspend and resume for agent fleets. In this tal ...

Awesome. [Music] Uh, hi everyone. My name is Tomas.I'm one of the co-founders of Graphite. Graphite is an AI code review uh, company. So to give some context on sort of where we see the industry right now and where we see it going.Software development currently and has always had two loops. The inner loop which is focused on development and the outer loop that's focused on review. Developers spend time in the inner loop.They get their code working. They get the feature the way they want it and then they go ...