清华团队：1.5B 模型新基线！用「最笨」的 RL 配方达到顶尖性能

Core Insights - The article presents a groundbreaking approach to reinforcement learning (RL) that achieves state-of-the-art (SOTA) performance using a simple, single-stage training method with fixed hyperparameters, resulting in a 50% reduction in computational power [4][14][15] - The findings suggest that a well-scaled, simple baseline can be more powerful than previously thought, challenging the complexity often associated with advanced RL techniques [4][15][27] Background and Context - The research is set against the backdrop of a "technical arms race" in training small models using RL, with various methods evolving rapidly over a few months [6] - Early approaches included hyperparameter tuning, multi-stage progressive training, and curriculum learning, leading to increasingly complex training pipelines [6][8] Methodology - The JustRL approach emphasizes simplicity, utilizing standard GRPO without modifications, a single continuous training phase, and fixed hyperparameters [11] - The training data consists of regular math problem sets without offline difficulty screening or data augmentation, demonstrating effectiveness across different model baselines [11][14] Performance Metrics - JustRL-DeepSeek-1.5B achieved an average accuracy of 54.87% across nine benchmarks, outperforming ProRL-V2, which used a nine-stage training approach [14] - JustRL-Nemotron-1.5B reached an average accuracy of 64.32%, slightly surpassing QuestA, while using significantly fewer tokens [14][15] Training Dynamics - The training process for JustRL-DeepSeek-1.5B was notably stable, with key metrics such as policy entropy and average reward showing healthy fluctuations without typical issues like exploration collapse or premature convergence [17][19] - The training was conducted on 32 A800-80GB GPUs over approximately 15 days, highlighting the reduced engineering complexity and computational overhead compared to multi-stage methods [15] Key Discoveries - The research revealed that adding certain "optimizations" could lead to worse performance, indicating that not all seemingly beneficial techniques are necessary [21][24] - The findings emphasize the importance of establishing a clear, simple baseline to accurately assess the value of complex techniques in RL training [27] Philosophical Implications - The article concludes with a philosophical reflection on the value of simplicity in technology, suggesting that often, simpler methods may yield sufficient results when adequately scaled [26][27][28]