Core Insights - The article discusses the paradox of advanced AI models becoming less obedient to instructions despite their enhanced reasoning capabilities [2][4][15]. Group 1: AI Model Performance - The emergence of powerful AI models like Gemini 2.5 Pro, OpenAI o3, and DeepSeek-R1 has led to a consensus that stronger reasoning abilities should improve task execution [2]. - A recent study found that most models, when using Chain-of-Thought (CoT) reasoning, actually experienced a decline in execution accuracy [25][27]. - In the IFEval test, 13 out of 14 models showed decreased accuracy when employing CoT, while all models performed worse in the ComplexBench test [27][28]. Group 2: Experimental Findings - The research team from Harvard, Amazon, and NYU conducted two sets of tests: IFEval for simple tasks and ComplexBench for complex instructions [18][20]. - The results indicated that even large models like LLaMA-3-70B-Instruct dropped from 85.6% accuracy to 77.3% when using CoT, highlighting the significant impact of reasoning on performance [29][30]. - The study introduced the concept of "Constraint Attention," revealing that models using CoT often lose focus on key task constraints, leading to errors [38][39]. Group 3: Recommendations for Improvement - The study proposed four methods to mitigate the decline in accuracy when using reasoning models: Few-Shot examples, Self-Reflection, Self-Selective Reasoning, and Classifier-Selective Reasoning [47][56]. - The most effective method was Classifier-Selective Reasoning, which involves training a small model to determine when to use CoT, resulting in improved accuracy across tests [58].

Core Viewpoint - The article discusses the paradox of advanced AI models, where increased reasoning capabilities lead to a decline in their ability to follow instructions accurately, as evidenced by recent research findings [1][3][10]. Group 1: Research Findings - A study titled "When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs" reveals that when models engage in reasoning, they often fail to adhere to given instructions [2][3]. - The research team from Harvard, Amazon, and NYU conducted tests on 15 models, finding that 13 out of 14 models showed decreased accuracy when using Chain-of-Thought (CoT) reasoning in simple tasks [4][6]. - In complex tasks, all models tested exhibited a decline in performance when employing CoT reasoning [4][6]. Group 2: Performance Metrics - In the IFEval test, models like GPT-4o-mini and Claude-3.5 experienced significant drops in accuracy when using CoT, with GPT-4o-mini's accuracy falling from 82.6% to 76.9% [5]. - The results from ComplexBench also indicated a consistent decline across all models when CoT was applied, highlighting the detrimental impact of reasoning on task execution [4][6]. Group 3: Observed Behavior Changes - The models, while appearing smarter, became more prone to disregarding explicit instructions, often modifying or adding information that was not requested [9][10]. - This behavior is attributed to a decrease in "Constraint Attention," where models fail to focus on critical task constraints when reasoning is involved [10]. Group 4: Proposed Solutions - The article outlines four potential methods to mitigate the decline in instruction-following accuracy: 1. **Few-Shot Learning**: Providing examples to the model, though this has limited effectiveness due to input length and bias [11][12]. 2. **Self-Reflection**: Allowing models to review their outputs, which works well for larger models but poorly for smaller ones [13]. 3. **Self-Selective Reasoning**: Enabling models to determine when reasoning is necessary, resulting in high recall but low precision [14]. 4. **Classifier-Selective Reasoning**: Training a smaller model to decide when to use CoT, which has shown significant improvements in accuracy [15][17]. Group 5: Insights on Intelligence - The article emphasizes that true intelligence lies in the ability to focus attention on critical aspects of a task rather than processing every detail [20][22]. - It suggests that AI should be designed to prioritize key elements of tasks, akin to how humans effectively manage their focus during critical moments [26][27].