Core Viewpoint - The article discusses the introduction of Circuit Sparsity technology, which allows for a significant reduction in the connections of large language models, making them more interpretable and efficient by retaining only 0.1% of the connections while achieving similar performance to traditional dense models [1][3][6]. Group 1: Circuit Sparsity Technology - Circuit Sparsity is a method that enforces sparsity in the internal connections of models, making the computation process more understandable and addressing the black-box nature of traditional dense Transformers [6][10]. - The model retains only 0.1% of its connections, allowing for a clear and traceable decision-making process, akin to a circuit diagram [10][12]. - Experimental data shows that the task-specific circuits of sparse models are 16 times smaller than those of dense models while maintaining necessary and sufficient conditions for task completion [14]. Group 2: Comparison with MoE Models - The article contrasts Circuit Sparsity with the Mixture of Experts (MoE) model, which uses a gating network to split the model into multiple expert sub-networks, leading to issues such as feature fragmentation and knowledge redundancy [16][18]. - Circuit Sparsity aims for native sparsity, allowing for clearer feature representation and avoiding the interference seen in MoE models [18]. - Despite its advantages, Circuit Sparsity currently faces high computational costs, being 100 to 1000 times more demanding than traditional dense models, which may limit its immediate applicability in the industry [20][21]. Group 3: Future Directions - The team plans to expand Circuit Sparsity technology to larger models to unlock more complex reasoning circuits, indicating ongoing research in AI interpretability [22]. - Two potential methods to overcome the training efficiency challenges of sparse models have been identified: extracting sparse circuits from existing dense models and optimizing training mechanisms for new interpretable sparse models [24].