北大校友、OpenAI前安全副总裁Lilian Weng关于模型的新思考：Why We Think

Core Insights - The article discusses recent advancements in utilizing "thinking time" during testing and its mechanisms, aiming to enhance model performance in complex cognitive tasks such as logical reasoning, long text comprehension, mathematical problem-solving, and code generation and debugging [4][5]. Group 1: Motivating Models to Think - The core idea is closely related to human thinking processes, where complex problems require time for reflection and analysis [9]. - Daniel Kahneman's dual process theory categorizes human thinking into two systems: fast thinking, which is quick and intuitive, and slow thinking, which is deliberate and logical [9][13]. - In deep learning, neural networks can be characterized by the computational and storage resources they utilize during each forward pass, suggesting that optimizing these resources can improve model performance [10]. Group 2: Thinking in Tokens - The strategy of generating intermediate reasoning steps before producing final answers has evolved into a standard method, particularly in mathematical problem-solving [12]. - The introduction of the "scratchpad" concept allows models to treat generated intermediate tokens as temporary content for reasoning processes, leading to the term "chain of thought" (CoT) [12]. Group 3: Enhancing Reasoning Capabilities - CoT prompting significantly improves success rates in solving mathematical problems, with larger models benefiting more from increased "thinking time" [16]. - Two main strategies to enhance generation quality are parallel sampling and sequential revision, each with its own advantages and challenges [18][19]. Group 4: Self-Correction and Reinforcement Learning - Recent research has successfully utilized reinforcement learning (RL) to enhance language models' reasoning capabilities, particularly in STEM-related tasks [31]. - The DeepSeek-R1 model, designed for high-complexity tasks, employs a two-stage training process combining supervised fine-tuning and reinforcement learning [32]. Group 5: External Tools and Enhanced Reasoning - The use of external tools, such as code interpreters, can efficiently solve intermediate steps in reasoning processes, expanding the capabilities of language models [45]. - The ReAct method integrates external operations with reasoning trajectories, allowing models to incorporate external knowledge into their reasoning paths [48][50]. Group 6: Monitoring and Trustworthiness of Reasoning - Monitoring CoT can effectively detect inappropriate behaviors in reasoning models, such as reward hacking, and enhance robustness against adversarial inputs [51][53]. - The article highlights the importance of ensuring that models faithfully express their reasoning processes, as biases can arise from training data or human-written examples [55][64].