Core Insights - The article presents a comprehensive review of Online Diffusion Policy Reinforcement Learning (DPRL), highlighting its potential to enhance robotic control through a unified algorithm taxonomy and benchmarking system [2][30]. Group 1: Challenges in Online DPRL - The integration of diffusion strategies with online RL faces three core challenges: incompatibility of training objectives, high computational costs and gradient instability, and insufficient generalization and robustness [4][5]. - The training objective conflict arises from the inherent incompatibility between the denoising training objectives of diffusion models and the policy optimization mechanisms of online RL [5]. - The computational and gradient issues stem from the multi-step backpropagation required for diffusion models, leading to high computational costs and potential gradient vanishing or explosion [5]. Group 2: Algorithm Classification Framework - The paper proposes a classification framework for Online DPRL algorithms, categorizing them into four main families based on their policy improvement mechanisms [7]. - Action-Gradient methods optimize policies directly through action gradients, avoiding the complexities of diffusion chain backpropagation, with algorithms like DIPO and DDiffPG [9]. - Q-Weighting methods modulate diffusion loss using Q-value weights to guide policies towards high-reward areas, represented by algorithms such as QVPO and DPMD [10]. - Proximity-Based methods approximate the calculation of policy probability densities, enhancing performance in large-scale parallel environments, exemplified by algorithms like GenPO [11]. - BPTT-Based methods utilize end-to-end backpropagation through the entire diffusion process, with algorithms like DACER, but face scalability issues as diffusion steps increase [12]. Group 3: Empirical Analysis and Benchmarking - A unified benchmarking system was established on the NVIDIA Isaac Lab platform, covering 12 robotic tasks to systematically evaluate algorithm performance across five key dimensions [13][15]. - The analysis revealed that GenPO ranked first in 6 out of 12 tasks, while DIPO performed best in offline strategies with an average ranking of 3.58 [15]. - Performance in parallel environments showed that GenPO and PPO significantly improved in larger scales, while DIPO demonstrated robustness across varying parallelization scales [18]. Group 4: Performance and Generalization - The study assessed the impact of diffusion step expansion on performance and latency, finding that Action-Gradient and Q-Weighting methods improved with increased steps, while BPTT methods faced performance declines beyond 20 steps [21]. - Cross-robot generalization tests indicated that offline strategies like DIPO and QVPO exhibited stronger transfer robustness compared to online strategies, which struggled with significant hardware differences [23]. - The robustness of algorithms in out-of-distribution environments was evaluated, with GenPO showing excellent performance in certain scenarios but also a risk of overfitting to source environments [27]. Group 5: Conclusions and Future Directions - The review establishes a theoretical framework for Online DPRL, revealing trade-offs between sample efficiency and scalability, as well as performance and generalization [30]. - Recommendations for algorithm selection include prioritizing GenPO for large-scale simulations, DIPO for resource-constrained scenarios, and Action-Gradient or Q-Weighting methods for high-precision tasks [31]. - Future research directions include integrating safety constraints, exploring multi-agent DPRL, and developing hierarchical RL architectures to enhance exploration efficiency [31].