OpenAI又Open了一下：发布可解释性新研究，作者来自Ilya超级对齐团队

Core Insights - OpenAI has introduced a new method for training smaller models that enhances interpretability, making the internal mechanisms of models easier for humans to understand [5][6][7] - The research focuses on creating sparse models with many neurons but fewer connections, simplifying neural networks for better comprehension [7][11] Summary by Sections Model Interpretability - OpenAI's language models have complex structures that are not fully understood, and the new method aims to bridge this gap [6] - The core idea is to train sparse models that maintain a high number of neurons while limiting their connections, making them simpler and more interpretable [7][11] Research Methodology - The researchers designed a series of simple algorithmic tasks to evaluate the model's interpretability, identifying the "circuit" for each task [13][18] - A "circuit" is defined as the smallest computational unit that allows the model to perform a specific task, represented as a graph of nodes and edges [15][16] Example of Circuit - An example task involves predicting the correct closing quote for a string in Python, demonstrating how the model can remember the type of opening quote to complete the string [19][22] Findings and Implications - The research indicates that larger, sparser models can produce increasingly powerful functions while maintaining simpler circuits [26] - This suggests potential for extending the method to understand more complex behaviors in models [27] Current Limitations - The study acknowledges that sparse models are significantly smaller than state-of-the-art models and still contain many "black box" elements [30] - Training efficiency for sparse models is currently low, with two proposed solutions: extracting sparse circuits from existing dense models or developing more efficient training techniques [31][32]