Core Insights - The report from CICC highlights that the short-term layered architecture remains mainstream due to engineering controllability, while VLA shows potential in complex tasks and human-machine interaction. The world model is viewed as a long-term direction due to its cross-device transfer capability [1] Group 1: Embodied Intelligence Algorithms - Layered control serves as the foundational architecture paradigm, utilizing a two-tier structure for engineering [1] - The VLA paradigm, based on VLM, enhances generalization and interaction capabilities, representing an active research direction [1] - The world model provides physical constraints through environmental modeling and future predictions, currently in the research-led stage [1] Group 2: Embodied Intelligence Data - Robotic data encompasses multimodal sources, with industries seeking low-cost data acquisition and high-efficiency application paths [2] - Data acquisition methods include real machines, video (first-person/third-person), and simulation [2] - Data security is a critical baseline, with humanoid robot manufacturers facing challenges related to permission isolation, data encryption systems, and cross-border transmission policies [2] Group 3: Hot Topics in Embodied Intelligence - The Scaling Law for robots has not yet seen explosive breakthroughs, with limitations such as insufficient real data production capacity and Sim2Real transfer being key constraints [3] - Benchmarking is driving the standardization of evaluation processes, as embodied robots lack a recognized quantitative framework [3] - Physical AI, which integrates physical knowledge with AI models, has progressed to applications in robotic operations [3]

AI Model Evaluation Landscape - Traditional benchmark tests are losing credibility due to "data leakage" and "score manipulation" [1] - LMArena platform uses "anonymous battles + human voting" to redefine the evaluation criteria for large models [1] - Top models from GPT to Claude, Gemini to DeepSeek are competing on LMArena [1] LMArena's Challenges - LMArena faces challenges to its fairness due to Meta's "ranking manipulation" incident, data asymmetry issues, and platform commercialization [1] - "Human judgment" in LMArena may contain biases and loopholes [1] Future of AI Evaluation - The industry is moving towards "real combat" Alpha Arena and a combination of "static and dynamic" evaluations [1] - The ultimate question is not "who is stronger," but "what is intelligence" [1]

Investment & Strategy - BMW's investment in its new architecture is the largest single investment ever made, exceeding €10 billion last year [1][3] - The company views this investment as a self-fulfilling prophecy, with widespread internal support [2] - BMW anticipates a profitable future from this platform, with iX3 deliveries expected in the first quarter of 2026 [3] Market & Competition - BMW aims to increase its global market share with its new electric car, alongside its existing combustion engine and plug-in hybrid X3 models [4] - The company believes its new car will be the industry benchmark in terms of performance and technology [5] - BMW expects to gain market share from Tesla, citing Tesla's sales decline in Europe, including double-digit drops in Germany and France [5][6] Technology & Performance - BMW emphasizes the first-mover advantage of its new car, highlighting its charging speed, range, and energy efficiency of 15 kilowatt hours per 100 kilometers [6][7] - The company believes its brand strength and high-quality recognition will give it an advantage over new entrants in the electric car market [8] Brand Perception & Customer Shift - BMW believes the pendulum is swinging back from Tesla to BMW, noting that in Europe, BMW already sells more electric cars than its competitor [8][9] - A 2019 survey indicated that many drivers were switching from BMW to Tesla Model 3s, but the trend is now reversing [8]

Core Problem & Solution - Traditional software development lifecycle is insufficient for AI applications due to non-deterministic models, requiring a data science approach and continuous experimentation [3] - The key is to reverse engineer metrics from real-world scenarios, focusing on product experience and business outcomes rather than abstract data science metrics [6] - Build evaluations (evals) at the beginning of the process, not at the end, to identify failures and areas for improvement early on [14] - Continuous improvement of evals and solutions is necessary to reach a baseline benchmark for optimization [19] Evaluation Methodology - Evaluations should mimic specific user questions and criteria relevant to the solution's end goal [7] - Use Large Language Models (LLMs) to generate evaluations, considering different user personas and expected answers [9][11] - Focus on the details of each evaluation failure to understand the root cause, whether it's the test definition or the solution's performance [15] - Experimentation involves changing models, logic, prompts, or data, and continuously running evaluations to catch regressions [16][18] Industry Specific Examples - For customer support bots, measure the rate of escalation to human support as a key metric [5] - For text-to-SQL or text-to-graph database applications, create a mock database with known data to validate expected results [22] - For call center conversation classifiers, use simple matching to determine if the correct rubric is applied [23] Key Takeaways - Evaluate AI applications the way users actually use them, avoiding abstract metrics [24] - Frequent evaluations enable rapid progress and reduce regressions [25] - Well-defined evaluations lead to explainable AI, providing insights into how the solution works and its limitations [26]

Overview - DDN Infinia 在 Oracle Cloud Infrastructure (OCI) 上的性能展示，但强调这仅为技术预览，未来可能发生变化 [1] - Infinia 架构提供广泛的数据管理能力，包括多种数据 IO 路径、核心存储组件（如 scale-out KV 存储、always-on 加密和数据缩减）、原生多租户等 [2] - Infinia 完全软件定义和容器化，可在物理或虚拟化硬件上运行，适用于云部署 [2] Technical Details & Performance - 在 OCI 内部测试使用了 6 个 BM dense ioe5 计算实例作为 Infinia 集群的主机，以及 6 个 BM standard E5.192% 实例作为客户端，客户端实例具有单个 100 GB 连接 [2] - 在 dense ioe5 实例中，Infinia 软件仅使用了 128 个可用核心中的 32 个 [2] - 使用 warp 在分布式基准测试模式下进行 IO 生成，确保每个客户端并发地向所有 Infinia 集群节点发送操作，并在所有客户端和所有 Infinia 集群节点之间创建完整的 IO 网格 [3] - Put 操作的吞吐量约为 28 GB/s 到 30 GB/s，每个客户端和每个 Infinia 节点平均处理速度约为 4800 MB/s (约 5 GB/s) [5] - Get 操作的吞吐量约为 35 GB/s 到 37.5 GB/s，负载均匀分布在所有客户端和 Infinia 节点上，约为 6100 MB/s (约 6.5 GB/s) [6] - 实现了 5 毫秒的 time to first byte，对于 S3 对象 IO 来说非常出色 [6] Conclusion - 软件定义的 Infinia 不仅可以在云中的 Oracle 计算基础设施上运行，而且即使是小型六节点集群也能实现出色的性能 [7]

Market Trends - The S&P 500 has decreased by 85% when measured in Bitcoin terms since 2020 [1] - Bitcoin is suggested as the new benchmark for measurement [1]

Vector Database Benchmarking Challenges - The vector database market is filled with misleading benchmarks, where every database claims to be both faster and slower than its competitors [1] - Meaningful vector search benchmarks are uniquely tricky to build [1] - It is crucial to tailor benchmarks to specific use cases to get useful results [1] - Benchmarks should be tweaked and verified independently to avoid blindly trusting marketing claims [1] Recommendations for Benchmarking - Avoid trusting glossy charts and marketing materials when evaluating vector databases [1] - Build meaningful benchmarks tailored to specific use cases to get accurate performance assessments [1] - Independently verify and tweak benchmarks to ensure they reflect real-world performance [1] About the Speaker - Philipp Krenn leads Developer Relations at Elastic, the company behind Elasticsearch, Kibana, Beats, and Logstash [1]