作者 | 我要吃鸡腿 编辑 | 大模型之心Tech 原文链接： https://zhuanlan.zhihu.com/p/1963658684765833212 点击下方 卡片 ，关注" 大模型之心Tech "公众号 戳我-> 领取大模型巨卷干货 在深入探讨 LLaVA 和 Qwen3-VL 的具体实现之前，我们必须先搭建一个稳固的认知框架。幸运的是，尽管实现细节千差万别，当前绝大多数主流的 多模态大模型都遵循着一个共同的、优雅的"三位一体"黄金架构。我们可以将其生动地比喻为为 AI 打造一套完整的"感知-思考"系统： AI 的"眼睛" (视觉编码器) : 负责最前端的感知。它的任务是将输入的像素世界——无论是静态图片还是动态视频，转化为机器能够理解的、蕴含 丰富语义的数学表达（即特征向量）。 本文只做学术分享，已获转载授权 ，欢迎添加小助理微信AIDriver004做进一步咨询 引言：当 AI 睁开双眼，我们看到了一个怎样的未来？ 曾几何时，我们对人工智能的印象还停留在那个聪慧但略显"盲目"的"数字大脑"上——它能写诗、能编程、能回答深奥的哲学问题，但这一切都局限 于冰冷的文本世界。然而，就在最近两年，一场 ...

作者 | 我要吃鸡腿 编辑 | 大模型之心Tech 原文链接： https://zhuanlan.zhihu.com/p/1963658684765833212 本文只做学术分享，已获转载授权 ，欢迎添加小助理微信AIDriver004做进一步咨询 点击下方 卡片 ，关注" 大模型之心Tech "公众号 戳我-> 领取大模型巨卷干货 引言：当 AI 睁开双眼，我们看到了一个怎样的未来？ 曾几何时，我们对人工智能的印象还停留在那个聪慧但略显"盲目"的"数字大脑"上——它能写诗、能编程、能回答深奥的哲学问题，但这一切都局限 于冰冷的文本世界。然而，就在最近两年，一场深刻的变革正在悄然发生。 您或许已经惊叹于 GPT-5 那般流畅自如的实时图片对话，它能"看到"您房间的布局并给出整理建议；又或者，您可能对 Qwen3-VL 直接"注视"着手 机屏幕、精准地点击按钮、操作应用程序的能力感到不可思议。AI 不再仅仅是一个"只会读书"的语言模型，它正在进化成一个能听、会看、可交互 的"智能体"，真正地睁开了双眼，开始感知和理解我们所处的这个五彩斑斓的物理世界。 这场从"符号"到"感知"的飞跃，背后究竟隐藏着怎样的技术密码 ...

Core Insights - The article discusses the evolution of artificial intelligence (AI) from a text-based model to a multimodal large model (MLLM) capable of perceiving and interacting with the physical world through vision and language [3][4]. - It highlights two successful technical evolution paths in MLLM: the LLaVA series, which emphasizes simplicity, and the Qwen3-VL, which focuses on deep integration [3][4]. Group 1: MLLM Architecture - MLLM follows a "trinity" architecture consisting of a visual encoder (Vision Transformer), a language model (LLM), and a connector that facilitates communication between the two [6][10]. - The visual encoder transforms images into mathematical representations, while the LLM processes these representations to generate coherent text responses [10][22]. - The connector acts as a bridge, translating visual features into a format that the LLM can understand, ensuring seamless integration of visual and textual information [36][37]. Group 2: Vision Transformer (ViT) - ViT revolutionizes image processing by treating images as sequences of patches, allowing the model to leverage transformer architecture for visual understanding [11][13]. - The process involves segmenting images into patches, flattening them into vectors, and adding positional information to maintain spatial context [13][16]. - ViT's multi-head attention mechanism enables the model to capture relationships between distant elements in an image, enhancing its ability to understand complex visual scenes [21][22]. Group 3: Language Model (LLM) - LLM serves as the cognitive core of MLLM, integrating visual and textual information to generate contextually relevant responses [22][23]. - The input to LLM is a combined sequence of visual and language tokens, allowing for a comprehensive understanding of the context [24][25]. - LLM employs autoregressive generation to predict the next token based on the entire context, facilitating coherent and contextually appropriate outputs [26][30]. Group 4: Connector Design - The connector's design is crucial for bridging the gap between visual and textual modalities, with two main approaches: the minimalist approach of LLaVA and the more complex Q-Former used in BLIP-2 [38][40]. - LLaVA's connector is a simple linear transformation that relies on the LLM's strength to learn the mapping between modalities [40][41]. - Q-Former, on the other hand, actively extracts and refines key information from visual features before passing them to the LLM, enhancing efficiency and reducing computational load [42][53]. Group 5: Challenges and Solutions - The article addresses the challenge of processing high-resolution images without overwhelming the model's computational capacity, leading to the exploration of different design philosophies [64]. - LLaVA's AnyRes solution allows the model to handle images of arbitrary resolutions by focusing on preprocessing techniques rather than restructuring the model [65].