Investment & Strategy - BMW's investment in its new architecture is the largest single investment ever made, exceeding €10 billion last year [1][3] - The company views this investment as a self-fulfilling prophecy, with widespread internal support [2] - BMW anticipates a profitable future from this platform, with iX3 deliveries expected in the first quarter of 2026 [3] Market & Competition - BMW aims to increase its global market share with its new electric car, alongside its existing combustion engine and plug-in hybrid X3 models [4] - The company believes its new car will be the industry benchmark in terms of performance and technology [5] - BMW expects to gain market share from Tesla, citing Tesla's sales decline in Europe, including double-digit drops in Germany and France [5][6] Technology & Performance - BMW emphasizes the first-mover advantage of its new car, highlighting its charging speed, range, and energy efficiency of 15 kilowatt hours per 100 kilometers [6][7] - The company believes its brand strength and high-quality recognition will give it an advantage over new entrants in the electric car market [8] Brand Perception & Customer Shift - BMW believes the pendulum is swinging back from Tesla to BMW, noting that in Europe, BMW already sells more electric cars than its competitor [8][9] - A 2019 survey indicated that many drivers were switching from BMW to Tesla Model 3s, but the trend is now reversing [8]

Core Problem & Solution - Traditional software development lifecycle is insufficient for AI applications due to non-deterministic models, requiring a data science approach and continuous experimentation [3] - The key is to reverse engineer metrics from real-world scenarios, focusing on product experience and business outcomes rather than abstract data science metrics [6] - Build evaluations (evals) at the beginning of the process, not at the end, to identify failures and areas for improvement early on [14] - Continuous improvement of evals and solutions is necessary to reach a baseline benchmark for optimization [19] Evaluation Methodology - Evaluations should mimic specific user questions and criteria relevant to the solution's end goal [7] - Use Large Language Models (LLMs) to generate evaluations, considering different user personas and expected answers [9][11] - Focus on the details of each evaluation failure to understand the root cause, whether it's the test definition or the solution's performance [15] - Experimentation involves changing models, logic, prompts, or data, and continuously running evaluations to catch regressions [16][18] Industry Specific Examples - For customer support bots, measure the rate of escalation to human support as a key metric [5] - For text-to-SQL or text-to-graph database applications, create a mock database with known data to validate expected results [22] - For call center conversation classifiers, use simple matching to determine if the correct rubric is applied [23] Key Takeaways - Evaluate AI applications the way users actually use them, avoiding abstract metrics [24] - Frequent evaluations enable rapid progress and reduce regressions [25] - Well-defined evaluations lead to explainable AI, providing insights into how the solution works and its limitations [26]

Overview - DDN Infinia 在 Oracle Cloud Infrastructure (OCI) 上的性能展示，但强调这仅为技术预览，未来可能发生变化 [1] - Infinia 架构提供广泛的数据管理能力，包括多种数据 IO 路径、核心存储组件（如 scale-out KV 存储、always-on 加密和数据缩减）、原生多租户等 [2] - Infinia 完全软件定义和容器化，可在物理或虚拟化硬件上运行，适用于云部署 [2] Technical Details & Performance - 在 OCI 内部测试使用了 6 个 BM dense ioe5 计算实例作为 Infinia 集群的主机，以及 6 个 BM standard E5.192% 实例作为客户端，客户端实例具有单个 100 GB 连接 [2] - 在 dense ioe5 实例中，Infinia 软件仅使用了 128 个可用核心中的 32 个 [2] - 使用 warp 在分布式基准测试模式下进行 IO 生成，确保每个客户端并发地向所有 Infinia 集群节点发送操作，并在所有客户端和所有 Infinia 集群节点之间创建完整的 IO 网格 [3] - Put 操作的吞吐量约为 28 GB/s 到 30 GB/s，每个客户端和每个 Infinia 节点平均处理速度约为 4800 MB/s (约 5 GB/s) [5] - Get 操作的吞吐量约为 35 GB/s 到 37.5 GB/s，负载均匀分布在所有客户端和 Infinia 节点上，约为 6100 MB/s (约 6.5 GB/s) [6] - 实现了 5 毫秒的 time to first byte，对于 S3 对象 IO 来说非常出色 [6] Conclusion - 软件定义的 Infinia 不仅可以在云中的 Oracle 计算基础设施上运行，而且即使是小型六节点集群也能实现出色的性能 [7]

Market Trends - The S&P 500 has decreased by 85% when measured in Bitcoin terms since 2020 [1] - Bitcoin is suggested as the new benchmark for measurement [1]

Vector Database Benchmarking Challenges - The vector database market is filled with misleading benchmarks, where every database claims to be both faster and slower than its competitors [1] - Meaningful vector search benchmarks are uniquely tricky to build [1] - It is crucial to tailor benchmarks to specific use cases to get useful results [1] - Benchmarks should be tweaked and verified independently to avoid blindly trusting marketing claims [1] Recommendations for Benchmarking - Avoid trusting glossy charts and marketing materials when evaluating vector databases [1] - Build meaningful benchmarks tailored to specific use cases to get accurate performance assessments [1] - Independently verify and tweak benchmarks to ensure they reflect real-world performance [1] About the Speaker - Philipp Krenn leads Developer Relations at Elastic, the company behind Elasticsearch, Kibana, Beats, and Logstash [1]