Core Viewpoint - The article discusses the limitations of current Language Models (LM) in handling long texts and introduces Fourier Position Embedding (FoPE) as a solution to enhance the long text generalization capability of Transformer models [1][16]. Group 1: Limitations of RoPE - RoPE's periodic extension is limited by spectrum damage, which arises when each dimension of hidden states is assumed to have a single frequency, leading to confusion in semantic transmission [4][7]. - Spectrum damage is caused by three main factors: linear functions, activation functions, and time-domain truncation [7][16]. Group 2: Introduction of FoPE - FoPE aims to improve the frequency domain robustness and periodic extension of models, thereby enhancing long text generalization [16][17]. - The core idea of FoPE is to model each dimension as a Fourier series, allowing for the decoding of more frequency information despite the presence of spectrum damage [17]. Group 3: Experimental Results - The article presents experimental results showing that FoPE consistently outperforms RoPE across various benchmarks, particularly in long text tasks [18][19]. - For instance, in the GovReport benchmark, FoPE improved performance from 13.02 to 13.27 for lengths 0-4k, and from 11.35 to 12.50 for lengths 4-8k, demonstrating significant enhancements [19]. Group 4: Potential Applications - The findings and algorithms derived from Fourier analysis may have broader implications, potentially applicable in areas such as long video generation, multi-model collaboration, and even outside AI in semantic communication and brain-machine interfaces [21].