Core Viewpoint - The article discusses the defense of Zitong Yang's doctoral thesis on "Continually Self-Improving AI," highlighting the limitations of current AI models and proposing solutions for continuous self-improvement in AI systems [1][4]. Group 1: Research Directions - The first core research direction is "Synthetic Continuing Training," which utilizes entity graph synthesis data generation technology to enable models to continuously learn niche domain knowledge post-pretraining while avoiding catastrophic forgetting [4][28]. - The second direction explores self-improvement of pretraining capabilities through "Synthetic Guided Pretraining," allowing models to autonomously discover potential structures and relationships within vast documents, thereby optimizing their pretraining effectiveness and significantly reducing factual error rates [4][79]. - The third direction showcases the potential of "AI Designing AI," where an independent research environment is constructed that includes a codebase and value function, introducing evolutionary search mechanisms for models to autonomously propose algorithm ideas, write code, and run experiments [4][116]. Group 2: Limitations of Current AI Models - Current AI models face three major limitations: static weights post-training, reliance on limited human data for expansion, and dependence on human-discovered algorithms [16][21][27]. - The static nature of model weights after initial training prevents continuous knowledge acquisition and integration of new information without catastrophic forgetting [16]. - The reliance on finite human data limits the depth and breadth of knowledge that models can acquire, as the amount of available data is diminishing [21]. - Current AI systems are constrained by the algorithms that humans can discover, which are often labor-intensive and costly to develop [27]. Group 3: Synthetic Continuing Training - The goal of "Synthetic Continuing Training" is to teach language models knowledge from niche domains using synthetic data, addressing the sparsity of knowledge without such data [32][40]. - A dataset of 265 professional books, totaling approximately 1.8 million tokens, was used to evaluate the model's understanding of these documents through a closed-book question-answering task [41][46]. - The model's performance was benchmarked against static models, with Llama 3's base model achieving an accuracy of 39% in closed-book settings, while the introduction of synthetic data improved performance significantly [50][52]. Group 4: Self-Improvement of Pretraining Capabilities - The concept of "Synthetic Guided Pretraining" aims to enhance pretraining capabilities by leveraging cross-document correlations through synthetic data generation [79][81]. - The methodology involves pretraining a language model, fine-tuning it as a synthetic data generator, and then combining real and synthetic data for further pretraining to improve performance [81][99]. - Results indicated that models utilizing synthetic data showed significant improvements in performance metrics compared to those relying solely on repeated real data [104][109]. Group 5: AI Designing AI - The article introduces the concept of an "AI Research Environment," which abstracts the requirements for conducting AI experiments, allowing models to autonomously generate and evaluate ideas [116][124]. - This environment includes a codebase and a value function to assess the quality of generated ideas, facilitating a structured approach to AI research [124][126]. - The implementation of this environment demonstrated the potential for AI to contribute to its own development, achieving competitive results in various tasks [137][149].