作者 | 理查德·萨顿（Richard Sutton） 2）每一个学习到的权重，都配有一个专门的步长参数，该参数通过在线交叉验证进行元学习； 原标题 | OaK 架构：一个源于经验的超级智能构想 来源 | RLC 2025 会议文章 （ youtu.be/gEbbGyNkR2U ） 编译 | 王启隆 出品丨AI 科技大本营（ID：rgznai100） 随着人工智能发展成为一个庞大的产业，它在很大程度上已经迷失了方向。 我们需要什么才能重回正轨，去探寻真正的智能？ 我们需要能够持续学习的智能体、世界模型和规划能力，以及学习高层次知识和通过元学习掌握泛化的能力。 OaK 架构 正是对所有这些需求的一个系统性回应。从整体上看，它是一个基于模型的强化学习架构，并具备三个鲜明特点： 1）其所有组件都能持续学习； 3）状态和时间上的抽象概念，通过一个我们称之为 FC-STOMP 的五步演进路径被持续创造出来，即：特征构建（ F eature C onstruction）、 基于特征提出子任务（posing a S ub T ask）、学习一个选项来解决该子任务（learning an O ption）、学习该选项的模型（ ...

Core Viewpoint - Richard Sutton, the father of reinforcement learning and 2024 ACM Turing Award winner, presented a vision for achieving general artificial intelligence (AGI) and superintelligence through the OaK architecture, which is based on experiential learning and outlines a clear roadmap for AI development [2][4]. Group 1: OaK Architecture Overview - The OaK architecture is not a complete algorithm but a vision that breaks down the goals for AI development into eight necessary steps, highlighting the current gaps and potential development paths [2][6]. - Sutton emphasizes the importance of a simple and general AI agent architecture that learns from experience rather than relying on pre-defined domain knowledge [10][13]. Group 2: Key Concepts in OaK Architecture - The architecture focuses on "open-ended abstraction," allowing the agent to continuously develop its conceptual framework and understanding of the world without being limited by predefined knowledge [13][28]. - Sutton introduces two critical concepts: design time (before deployment) and runtime (during operation), advocating for learning based on experience during runtime to adapt to the complexities of the world [18][20]. Group 3: Learning and Decision-Making - The architecture proposes that agents should learn solely from runtime experiences, as the complexity of the world cannot be fully anticipated or pre-defined [30][31]. - Sutton argues that the agent's knowledge is inherently approximate due to the vast complexity of the world, necessitating a focus on learning and planning during runtime [37][38]. Group 4: Reinforcement Learning and Reward Hypothesis - The reinforcement learning framework is defined by the goal of maximizing a scalar reward signal, which is central to the agent's learning process [42][47]. - Sutton posits that even a simple reward signal can lead to the emergence of intelligent behavior in a sufficiently complex environment [51]. Group 5: Common Agent Model - The common model of intelligent agents includes components such as perception, value function, reactive policy, and transition model, which are interconnected to facilitate learning and planning [58][61]. - This model serves as a foundation for the OaK architecture, which seeks to enhance it by introducing higher-level abstractions and multiple value functions for different subproblems [67][72]. Group 6: Implementation Steps of OaK Architecture - The implementation of the OaK architecture involves eight parallel steps, including learning strategies for maximizing rewards, generating new state features, and constructing corresponding subproblems [82][85]. - Each step is contingent on the successful realization of continuous deep learning and the ability to generate and evaluate new features [86][90]. Group 7: Future Directions and Challenges - Sutton acknowledges that while some steps in the OaK architecture are feasible, significant challenges remain, particularly in achieving reliable continuous learning in nonlinear deep learning networks [89][96]. - The architecture aims to create a system that evolves through an open-ended cycle of exploration and learning, with the ultimate goal of enhancing the agent's ability to abstract and generalize from experiences [160].