Core Insights - The METR Institute's research indicates that experienced open-source developers took an average of 19% longer to complete tasks when using AI programming tools [1][4][9] - Developers initially believed that AI would enhance their efficiency, predicting a 24% increase in speed, but the actual data contradicted this perception [2][9] Experiment Design - The study utilized a randomized controlled trial (RCT) to assess the impact of AI tools in real-world settings, which is considered the most rigorous method for measuring causal relationships [4][19] - Sixteen senior developers were tracked, completing 246 actual tasks across various open-source projects, with tasks randomly assigned to either an AI tool group or a non-AI group [7][19] - The AI group primarily used Cursor Pro, which integrates major models like Claude 3.5 and Claude 3.7 Sonnet [7] Findings on Developer Behavior - AI users spent more time on tasks due to increased interactions with AI, such as prompt design, reviewing AI outputs, and waiting for responses, rather than actively coding [10][11][15] - Developers reported feeling they saved time, despite data showing they were slower, indicating a "fast illusion" stemming from the new workflow dynamics introduced by AI [10][16] Implications for AI Evaluation - The research challenges existing AI evaluation benchmarks, which often rely on isolated, artificially simplified tasks that do not reflect the complexities of real-world projects [18][19] - The findings suggest that the perceived efficiency gains from AI tools may be misleading, as they do not necessarily translate to improved productivity in complex tasks [21][23] - The study highlights the potential for AI tools to alter workflows rather than enhance efficiency, affecting attention distribution and the pace of work [23]