Core Insights - The article discusses the shift in focus from Transformer models to alternative approaches in AI research, as highlighted by Llion Jones, co-founder and CTO of Sakana AI, who is reducing his research time on Transformers and seeking new goals [1][3] - Jerry Tworek, former VP of research at OpenAI, has founded Core Automation, which aims to explore a different path in AI model development, specifically focusing on "Continual Learning" capabilities [6][10] - Core Automation is seeking $500 million to $1 billion in funding and plans to develop models that require significantly less data and computational resources compared to current leading models [11][16] Company Developments - Core Automation is in its early stages, with its funding and product direction still subject to change, but it represents a growing group of researchers advocating for a fundamental transformation in AI [8][9] - Tworek's vision includes a single algorithm named "Ceres," which contrasts with the typical multi-stage training process used by major companies [16] - The company aims to automate the production of its own products, with initial goals in industrial automation and long-term ambitions that include creating self-replicating factories and bio-machines [16] Industry Trends - The article notes a trend among researchers who believe that current mainstream model development techniques are inadequate for achieving significant breakthroughs in fields like biology and medicine [9] - There is a growing enthusiasm in the capital markets for new experimental labs, as evidenced by recent funding rounds for startups like Humans& and Thinking Machines Lab, despite many lacking revenue or products [15] - The exploration of "Continual Learning" is not exclusive to Core Automation, as other labs like Safe Superintelligence are pursuing similar goals [13][14]

Core Insights - The article discusses the breakthrough of Nested Learning by Google's DeepMind, which may address the long-standing issue of "catastrophic forgetting" in AI, potentially leading to advancements towards Artificial General Intelligence (AGI) [1][52] - Nested Learning is positioned as a successor to the Transformer architecture, suggesting a shift from passive training to active evolution in AI systems [1][2] Group 1: Nested Learning and AGI - Nested Learning is highlighted as a significant research focus for DeepMind, with predictions that it could lead to minimal AGI by 2028 with a 50% confidence level [7][9] - The concept of Nested Learning is described as a framework that allows AI to build associative memory, enabling continuous learning without the need for retraining [1][19] - Shane Legg, co-founder of DeepMind, emphasizes that there are no current blockers to achieving continual learning, indicating progress in this area [5][7] Group 2: Technical Aspects of Nested Learning - The HOPE architecture is introduced as a mechanism for implementing Nested Learning, which combines fast self-updating systems with slow, multi-timescale memory [6][8] - The article outlines the importance of memory architecture, attentional bias, retention mechanisms, and learning rules in designing effective AI models [20][21] - The Nested Learning framework is said to unify various existing attention mechanisms and optimizers, allowing for a more dynamic understanding of memory in AI [21][24] Group 3: Performance and Implications - The HOPE architecture has shown superior performance in tasks requiring long context and continual learning compared to existing models, indicating its potential effectiveness [33][47] - The article raises concerns about the implications of AI systems that can learn continuously, suggesting that they may develop preferences based on past experiences, which could lead to ethical considerations [52]

Core Argument - The over-reliance on Reinforcement Learning (RL) in AI development may be leading the industry away from achieving Artificial General Intelligence (AGI), as current models lack the ability to learn autonomously from experience like humans do [3][4]. Group 1: Skills Preconditioning Paradox - Current AI models depend on "pre-baked" skills, such as using Excel or browsing the web, which highlights their lack of general learning capabilities, indicating that AGI is not imminent [5]. - The approach of embedding specific skills into models contradicts the essence of human-like learning, which does not require extensive pre-training for every task [4][17]. Group 2: Insights from Robotics - The challenges in robotics stem from algorithmic issues rather than hardware limitations; if AI had human-like learning capabilities, robots would already be widely adopted without the need for repetitive training [6][13]. Group 3: Economic Implications of AI - The argument that "technology diffusion takes time" is seen as a self-comforting excuse; if models truly possessed human-like intelligence, they would be rapidly adopted by businesses due to lower risks and no training requirements [7][19]. - The disparity between the value created by global knowledge workers, amounting to trillions of dollars, and the significantly lower revenue generated by AI models indicates that these models have not yet reached the threshold to replace human workers [8][22]. Group 4: The Importance of Continual Learning - The key bottleneck for achieving AGI lies in the ability for "Continual Learning," rather than merely stacking RL computational power; true AGI may take another 10 to 20 years to realize [9][25]. - The process of solving the continual learning problem is expected to be gradual, similar to the evolution of context learning capabilities, and may not yield immediate breakthroughs [29][30].