训练加速1.8倍，推理开销降78%！精准筛选题目高效加速RL训练丨清华KDD

Core Insights - The article discusses the significant advancements in reasoning capabilities of large language models (LLMs) through reinforcement learning fine-tuning, particularly highlighting the high costs associated with inefficient training processes [1][2]. Group 1: Training Efficiency - Traditional training methods like "Uniform Sampling" waste computational resources by randomly selecting questions that do not provide effective learning signals [2]. - The "Dynamic Sampling" approach, while more efficient, still incurs high costs due to the need for extensive self-evaluation by the model [2][6]. - The proposed MoPPS framework aims to dynamically predict question difficulty without the expensive self-evaluation process, thus enhancing training efficiency [3][6]. Group 2: MoPPS Framework - MoPPS utilizes a lightweight Bayesian model to quickly estimate question difficulty, allowing for efficient selection of training data [8][10]. - The framework models each question as a "bandit" problem, using a Beta distribution to estimate success rates based on training feedback [9][10]. - MoPPS introduces a recursive update mechanism that improves difficulty estimation over time, adapting to the model's evolving capabilities [11][13]. Group 3: Performance Improvements - MoPPS has demonstrated a training speed increase of 1.6x to 1.8x while reducing inference costs by up to 78.46% compared to traditional methods [18][21]. - The framework has shown significant advantages across various reasoning tasks, achieving better performance with fewer computational resources [18][21]. - The correlation between predicted and actual question difficulty is high, validating the effectiveness of MoPPS in accurately estimating task challenges [25][29]. Group 4: Versatility and Future Applications - MoPPS is compatible with multiple reinforcement learning algorithms and can adapt to different sampling strategies, enhancing its applicability [26][28]. - The framework's ability to incorporate prior knowledge can further accelerate initial training phases, making it a versatile tool for large-scale model fine-tuning [28][31]. - The research indicates potential for broader applications in the reinforcement learning fine-tuning of larger models in the future [31].