大模型“记性差一点”反而更聪明！金鱼损失随机剔除token，让AI不再死记硬背

Core Viewpoint - The article introduces a new method called "Goldfish Loss" that allows large language models to avoid memorizing training data verbatim, thereby enhancing their ability to learn language patterns while reducing the risk of overfitting [1][4]. Group 1: Goldfish Loss Concept - Goldfish Loss encourages models to forget specific details by randomly omitting a small portion of tokens during loss calculation [3][6]. - This method prevents the model from reproducing the training data word-for-word, while still enabling it to generate coherent text [4][9]. - The approach utilizes a hashing-based masking strategy to ensure consistency in the tokens that are omitted during training [8][14]. Group 2: Comparison with Traditional Methods - Unlike traditional regularization methods like Dropout, which introduce noise randomly, Goldfish Loss employs a static masking technique to consistently omit the same tokens across training iterations [11][19]. - This consistency fundamentally prevents the model from memorizing complete training sequences, as it cannot piece together omitted tokens from different training instances [12][14]. Group 3: Experimental Results - Experiments demonstrated that in extreme scenarios, standard training led to the model memorizing 84 out of 100 articles, while Goldfish Loss resulted in no memorization [22][24]. - In standard training scenarios, Goldfish Loss also significantly reduced the model's tendency to reproduce training data verbatim [24]. - Performance tests indicated no systematic differences in overall capabilities between models trained with Goldfish Loss and those trained with standard loss methods [26]. Group 4: Implications and Considerations - The core of Goldfish Loss lies in ignoring certain tokens during gradient calculations, which may require the model to process more data to compensate for the omitted information, potentially affecting computational efficiency [28].