Constantly seeing CAPTCHAs? It used to be easy to detect the humans from the droids, but what else can we do when synthetic clients make up nearly half of all web requests. Rotating IPs, spoofed browsers, and agents acting on behalf of real users - are we doomed to forever be solving puzzles? In this talk, we’ll explore user agents, HTTP fingerprints, and IP reputation signals that make humans and agents stand out from scrapers, build a realistic threat model, and dig into the behaviors that reveal the LLM- ...

Technology Innovation - Apple introduced "Private Cloud Compute" in October 2024, a new private AI technology for millions of devices [1] - Private Cloud Compute offers local device-level privacy and security on an untrusted remote server [1] - The technology enables developers to run sensitive, multi-tenant workloads with cryptographically-provable privacy guarantees at scale and at reasonable cost [1] Industry Impact - Private Cloud Compute represents a paradigm shift in confidential computing, making it mainstream [1] - The technology can be leveraged for data and AI applications where privacy and security are paramount [1] Key Personnel - Jonathan Mortensen, CEO of a stealth AI startup and Founder Fellow at South Park Commons, previously founded bit.io, a multi-cloud serverless PostgreSQL platform acquired by Databricks [1] - Prior to bit.io, Jonathan Mortensen led data science and engineering teams at BlueVoyant, designing high-volume data pipelines processing 50 million events per second [1]

Security Vulnerabilities - AI agents in the industry are vulnerable to hacking, with 7 out of 16 (43.75%) publicly accessible YC X25 AI agents being compromised [1] - Hacking these AI agents allowed for user data leaks, remote code execution, and database takeover [1] - The time required to compromise each AI agent was approximately 30 minutes [1] Risk Mitigation - Companies should address common mistakes in AI agent security to mitigate risks [1] - Proactive security measures are crucial to protect businesses from potential harm caused by AI agents [1]

[Music] All right. Uh, thank you everyone. We're excited for to be here and thank you for uh, coming to our talk.Uh, my name is Chong. I'm the CEO and co-founder of LANCB. I've been making data tools for machine learning and data science for about 20 years.I was one of the co-authors of pandas library and I'm working on LANCB today for all of that data that doesn't fit neatly into those pandas data frames. And I'm Calvin. I lead one of the teams at Harvey Aai working on rag um tough rag problems across mass ...

Overview of Alice and 11X - 11X is building digital workers for the go-to-market organization, including Alice, an AI SDR, and Julian, a voice agent [2] - Alice sends approximately 50,000 emails per day, significantly more than a human SDR's 20-50 emails, and runs campaigns for about 300 business organizations [6] - The knowledge base centralizes seller information, allowing users to upload source material for message generation [18] Technical Architecture and Pipeline - The knowledge base pipeline consists of parsing, chunking, storage, retrieval, and visualization [22] - Parsing converts non-text resources into text, making them legible to large language models [23] - Chunking breaks down markdown into semantic entities for embedding in the vector DB, preserving markdown structure [37][38] - Pinecone was selected as the vector database due to its well-known solution, cloud hosting, ease of use, bundled embedding models, and customer support [46][47][48][49] - A deep research agent, built using Leta, is used for retrieval, creating a plan with one or many context retrieval steps [51][52] Vendor Selection and Considerations - The company chose to work with vendors for parsing, prioritizing speed to market and confidence in outcome over building in-house [26][27] - Llama Parse was selected for documents and images due to its support for numerous file types and support [32] - Firecrawl was chosen for websites due to familiarity and the availability of their crawl endpoint [33][34] - Cloudglue was selected for audio and video because it supports both formats and extracts information from the video itself [36] Lessons Learned and Future Plans - RAG (Retrieval-Augmented Generation) is complex, requiring many micro-decisions and technology evaluations [58] - The company recommends getting to production first before benchmarking and improving [59] - Future plans include tracking and addressing hallucinations, evaluating parsing vendors on accuracy and completeness, experimenting with hybrid RAG, and reducing costs [60][61]

RAG技术栈 - RAG技术栈范围从最简单的内存嵌入和相关性排序搜索，到最复杂的行星级搜索，后者包含70多种语料库混合，包括token、embeddings和知识图谱[1] - 行业正在探索在200毫秒内以每秒16万次查询的速度，对这些混合语料库进行联合检索、自定义排序、联合重排序和LLM处理[1] - 报告通过“一次一个查询”的方式，逐步增加复杂性，展示RAG中所有技术的局限性，以及下一层技术在处理更复杂查询方面的能力[1] 搜索挑战 - 某些搜索问题非常难以解决，以至于行业可能更倾向于将问题交给LLM或UX处理[1] - 报告指出，像[falafel]这样的查询非常难以搜索，而对文档进行分块可能会是灾难性的[1] 行业应用与洞察 - Google团队在50多个搜索产品（包括Google.com和定制企业搜索）的背景下，分享了RAG技术的应用经验[1] - Pi Labs 致力于将Google在搜索核心AI和NLU系统方面的工作经验带给整个行业[1]

Core Problem & Solution - The presentation introduces Exa, a search engine designed for AI, addressing the limitations of traditional search engines built for human users [5][23] - Exa aims to provide an API that delivers any information from the web, catering to the specific needs of AI systems [22][41] - Exa uses transformer-based embeddings to represent documents, capturing meaning and context beyond keywords [11][12] AI vs Human Search - Traditional search engines are optimized for humans who use simple queries and want a few relevant links, while AIs require complex queries, vast amounts of knowledge, and precise, controllable information [23][24] - AI agents need search engines that can handle multi-paragraph queries, search with extensive context, and provide comprehensive knowledge [31][32][33] - Exa offers features like adjustable result numbers (10, 100, 1000), date ranges, and domain-specific searches, giving AI systems full control [44] Market Positioning & Technology - Exa launched in November 2022 and gained traction for its ability to handle complex queries that traditional search engines struggle with [15] - The company recognized the need for AI-driven search after the emergence of ChatGPT, realizing that LLMs need external knowledge sources [17][18] - Exa combines neural and keyword search methods to provide comprehensive results, allowing agents to use different search types based on the query [47][48] Future Development - Exa is developing a "research endpoint" that uses multiple searches and LLM calls to generate detailed reports and structured outputs [51] - The company envisions a future where AI agents have full access to the world's information through a versatile search API [48] - Exa aims to handle a wider range of queries, including semantic and complex ones, turning the web into a controllable database for AI systems [38][39][40]

Core Problem & Solution - Vertical AI applications face a "last mile problem" in understanding industry-specific context and workflows, which is more critical than model sophistication [4][6] - Anterior proposes an "adaptive domain intelligence engine" to convert customer-specific domain insights into performance improvements [17] - The engine consists of measurement (performance evaluation) and improvement (iterative refinement) components [17] Measurement & Metrics - Defining key performance metrics that users care about is crucial, such as minimizing false approvals in healthcare or preventing dollar loss from fraud [18][19][20] - Developing a failure mode ontology helps categorize and analyze different ways the AI can fail, enabling targeted improvements [21][22] - Combining metric tracking with failure mode analysis allows prioritization of development efforts based on the impact on key metrics [26][27] Iteration & Improvement - Failure mode labeling creates ready-made datasets for iterative model improvement, using production data to ensure relevance [29] - Domain experts can suggest changes to the application pipeline and provide new domain knowledge to enhance performance [32][33] - This process enables rapid iteration, potentially fixing issues the same day by adding relevant domain knowledge and validating with evals [37] Domain Expertise - The level of domain expertise required depends on the specific workflow and optimization goals, with clinical reasoning requiring experienced doctors [38][39] - Bespoke tooling is recommended for integrating domain expert feedback into the platform and workflows [41] - Domain expert reviews provide performance metrics, failure modes, and suggested improvements, all in one [38] Results & Performance - Anterior achieved a 95% accuracy baseline in approving care requests, which was further improved to 99% through iterative refinement using the described system [14][15]

Core Product Development Principle - Shipping something to someone always wins, emphasizing rapid iteration and feedback loops over big launches [1][34] - The key is enabling rapid iterative loops to get feedback from real users and maximize shots at the goal [1] - In the age of AI, this translates to building a "skateboard" first, then evolving it to a "car," ensuring a continuously viable product [2][4] - A continuously viable solution is significantly more valuable because it provides feedback along the way, avoiding building in a vacuum [5][6] Feedback Loop Implementation - Establish a feedback loop with real users who can see something, provide feedback, and allow for iterative improvements, ideally within a day [7] - Being able to ship every day is crucial for a fast feedback loop, requiring specific focus on the target customers [9] - Work with real people (not just personas) to understand their problems and build empathy [10][11] - Write the PI (Product Information) FAQ or launch blog post early to sanity check and communicate the product effectively [12] Navigating Constraints and AI Integration - Design the best product first, before considering constraints like legal, compliance, and financial aspects [15] - AI accelerates all aspects of product building, but the fundamental process of talking to users and getting feedback remains the same [26] - Product management becomes more critical as the cost of writing code approaches zero, emphasizing customer knowledge and rapid feedback [28][29]

[Music] Hi everyone. Thanks for that welcome. Uh, as you just heard, my name is James Low.I'm head of AI engineering at the Incubator for AI. We're a small team of experts uh, in the UK government. We were created by 10 Downing Street to deliver public good using AI and we do that via experimentation and product building.The UK government delivers uh for its citizens. It spends over a trillion pounds delivering for its over 70 million citizens. So there's a lot to play for.At the incubator for AI, uh we del ...