打破视频推理「先看后想」惯性，实现真正的「边看边想」丨CVPR'26

Core Insights - The article discusses the limitations of current large vision-language models (VLMs) in real-time video analysis, emphasizing the need for a shift from "frame-text interleaving" to "parallel" processing for effective real-time reasoning [1][4]. Group 1: Current Limitations of VLMs - Existing VLMs follow a sequential logic that is effective for offline tasks but leads to uncontrollable delays and evidence mismatch in streaming video scenarios [7][8]. - The "frame-text interleaving" approach, while improving real-time perception, still operates in a serial manner, resulting in low computational efficiency [9][10]. - Complex video understanding often requires Chain-of-Thought (CoT) reasoning, which significantly extends inference time and hinders real-time application [12][13]. Group 2: Proposed Solutions by TaYS - The TaYS framework introduces three key innovations: 1. Streaming attention masks to ensure true temporal causality, allowing the model to only access frames that have arrived [18][19]. 2. Decoupled positional encoding to separate "temporal order" from "thinking order," enhancing stability in temporal reasoning [20][21]. 3. Dual KV-Caches that enable visual encoding and text reasoning to run in parallel, significantly reducing both first token generation time (TTFT) and overall latency [22][23]. Group 3: Experimental Results - TaYS demonstrates superior accuracy in dynamic event reasoning, causal inference, and thematic understanding compared to batch processing and naive interleaved baselines [25]. - The framework achieves a substantial reduction in TTFT and overall latency, making it more efficient and reliable for real-time applications [26][27]. - The ablation studies confirm that parallel processing is crucial for maintaining low latency and accurate temporal understanding [27]. Group 4: Implications for Future Applications - TaYS represents a paradigm shift towards real-time intelligent applications, enabling smoother interactions in robotics, security monitoring, and live education [29][30][31]. - The framework allows models to "think in real-time," enhancing their applicability in various fields [33].