ACL 2025｜驱动LLM强大的过程级奖励模型（PRMs）正遭遇「信任危机」？

Core Insights - Large Language Models (LLMs) have shown remarkable capabilities in complex reasoning tasks, largely due to the empowerment of Process-Level Reward Models (PRMs) [1] - A recent study has revealed significant shortcomings in existing PRMs, particularly in identifying subtle errors during reasoning processes, raising concerns about their reliability [2] - The need for effective supervision of the reasoning process is emphasized, as current evaluation methods often overlook detailed error types in favor of final outcome correctness [3] PRMBench Overview - PRMBench is introduced as a comprehensive benchmark designed to evaluate the fine-grained error detection capabilities of PRMs, addressing the limitations of existing models [4] - The benchmark includes 6,216 carefully designed questions and 83,456 step-level fine-grained labels, ensuring depth and breadth in evaluating various complex reasoning scenarios [11] - PRMBench employs a multi-dimensional evaluation system focusing on simplicity, soundness, and sensitivity, further divided into nine subcategories to capture PRMs' performance on potential error types [11][25] Key Findings - The study systematically reveals deep flaws in current PRMs, with the best-performing model, Gemini-2-Thinking, scoring only 68.8, significantly below human-level performance of 83.8 [11][27] - Open-source PRMs generally underperform compared to closed-source models, highlighting reliability issues and potential training biases in practical applications [27] - The evaluation indicates that detecting redundancy in reasoning processes is particularly challenging for PRMs, marking it as a significant hurdle [27] Evaluation Metrics - PRMBench utilizes Negative F1 Score as a core metric to assess error detection performance, focusing on the accuracy of identifying erroneous steps [26] - The PRMScore combines F1 Score and Negative F1 Score to provide a comprehensive reflection of a model's overall capability and reliability [26] Implications for Future Research - The release of PRMBench serves as a wake-up call to reassess the capabilities of existing PRMs and accelerate the development of fine-grained error detection in complex reasoning scenarios [39] - PRMBench is expected to guide future PRM design, training, and optimization, contributing to the development of more robust and generalizable models [41]