AI大牛张祥雨：Transformer撑不起Agent时代

Core Insights - The current AI landscape, particularly in large models, is facing limitations due to the Transformer architecture, which is unable to effectively handle long-term memory and context processing [1][3][4] - Zhang Xiangyu, a prominent AI researcher, emphasizes that the existing Transformer models struggle with information flow and depth of understanding, particularly when processing sequences beyond 80,000 tokens [3][4] - There is a growing consensus among researchers that the Transformer architecture may have fundamental limitations, prompting a search for new breakthroughs in AI model design [4][5] Industry Trends - The AI industry appears to be in a "steady state," with many innovations converging around Transformer variants, yet these modifications do not fundamentally alter its modeling capabilities [3] - New architectures such as Mamba and TTT (Test-Time Training) are gaining attention, with major companies like Nvidia, Meta, and Tencent exploring their integration with Transformers [4] - Research institutions are also venturing into non-Transformer architectures, as evidenced by the development of the brain-like pulse model "Shunxi 1.0" by the Chinese Academy of Sciences [4] Future Directions - The team at Jumpshare is exploring new architectural directions, particularly focusing on non-linear recursive networks, although this presents challenges in system efficiency and parallelism [5] - The need for collaborative design in implementing these new architectures is highlighted as a critical factor for success in overcoming the limitations of current models [5]