扩散语言模型的潜力被严重低估了！新国立发现可全面超越自回归

Core Insights - The article discusses the emergence of Diffusion Language Models (DLM) as a new paradigm in language modeling, showcasing their ability to learn more effectively under data constraints compared to traditional Autoregressive (AR) models [2][5][6]. Research Background - Autoregressive language models are currently the mainstream in large-scale language modeling, but high-quality data has become a significant bottleneck for model expansion [3]. - In scenarios with limited data, the ability to extract more information from each unique token becomes crucial, indicating that data, rather than computation, is the limiting factor [4]. Crossover Phenomenon - The research defines a "Crossover" point where DLM models surpass AR models in performance under limited data conditions, demonstrating approximately three times the data efficiency of AR models [5]. - Factors influencing the timing of this crossover include data quantity and quality, as well as model size [8]. Experimental Results - Under lower data budgets, DLM significantly outperforms AR models, achieving comparable performance with fewer unique tokens [13]. - The quality of data also plays a critical role, with higher quality data delaying the crossover point for DLM models [16]. - Increasing model size leads to an earlier crossover point, as AR models quickly saturate under data constraints, while DLM continues to improve with larger sizes [19]. Computational Efficiency - DLM models consistently outperform AR models across various sparsity levels, with denser architectures yielding better performance, especially under data constraints [22]. - The introduction of noise in the training process enhances DLM's performance, while AR models struggle to maintain performance under high noise levels [26]. Large-Scale Token Training - The research validates the crossover phenomenon in large-scale unique token datasets, particularly in generation tasks, indicating that DLM retains significant untapped potential even after extensive training [31]. - The performance of DLM remains robust even with extreme data repetition, suggesting its ability to extract more information from fixed datasets [40]. Overfitting and Model Behavior - DLM may exhibit overfitting in scenarios with limited unique data and larger model sizes, but performance degradation typically occurs later in the training process [43]. - The absence of strict causal biases in DLM allows for better modeling of complex data patterns, enhancing its learning capabilities [44]. Future Directions - While DLM shows promise, challenges remain in ensuring data security and privacy, particularly in dense training scenarios, and the architecture for practical deployment is still less mature compared to AR models [46].