【导读】浙江大学ReLER团队开源ContextGen框架，攻克多实例图像生成中布局与身份协同控制难题。基于Diffusion Transformer架构，通过双重注意 力机制，实现布局精准锚定与身份高保真隔离，在基准测试中超越开源SOTA模型，对标GPT-4o等闭源系统，为定制化AI图像生成带来新突破。 在定制化AI图像生成领域，多实例图像生成（MIG）面临一个关键的协同控制挑战：精确布局控制和多主体身份保真的同步实现。 现有方法往往只能达成二者之一，少数能兼顾的方法在性能上也存在显著不足。 为解决这一布局与身份的协同控制瓶颈，浙江大学ReLER团队提出了ContextGen框架，首次在Diffusion Transformer (DiT) 架构内部，通过双重上下文注意 力机制实现了架构级的分层解耦控制。 ContextGen在基准测试上，身份保持能力超越SOTA开源模型，并成功对标了GPT-4o和Nano-Banana等强大的闭源系统，实现了在复杂定制化控制方面实现 了关键突破。 论文地址：https://arxiv.org/abs/2510.11000 代码地址：https://github.com/n ...

Core Insights - The article discusses the advancements in image generation, particularly focusing on the challenges in Multi-Instance Image Generation (MIG), which include layout control and identity preservation [2][5]. Group 1: ContextGen Framework - ContextGen is introduced as a new framework based on Diffusion Transformer (DiT) aimed at addressing the challenges of layout control and identity preservation in MIG tasks [5][6]. - The framework employs a dual-core mechanism that operates on a unified context token sequence, enhancing both layout and identity fidelity [8][10]. Group 2: Mechanisms of ContextGen - The Contextual Layout Anchoring (CLA) mechanism focuses on global context guidance, utilizing user-designed or model-generated layout images to ensure precise global layout control and initial identity information [10]. - The Identity Consistency Injection (ICA) mechanism injects identity information from high-fidelity reference images into corresponding target locations, ensuring consistency across multiple instances [12]. Group 3: Data Foundation - The IMIG-100K dataset is introduced as the first large-scale, detailed annotated dataset designed for image-guided multi-instance generation tasks, providing various difficulty levels and detailed layout and identity annotations [14]. Group 4: Performance Optimization - ContextGen incorporates a reinforcement learning phase based on preference optimization (DPO) to encourage creativity and diversity in generated images, moving beyond rigid replication of layout content [17]. Group 5: Experimental Validation - ContextGen demonstrates superior performance in quantitative and qualitative evaluations, surpassing all open-source models and matching closed-source commercial models in identity consistency [21][25]. - In the LAMICBench++ benchmark, ContextGen achieved an average score improvement of +1.3% over existing open-source models, showcasing its capabilities in complex multi-instance scenarios [21]. Group 6: User Interaction - A user-friendly front-end interface is included in the project, allowing users to upload reference images, add new materials via text, and design layouts through drag-and-drop functionality [32]. Group 7: Future Outlook - The ReLER team plans to further optimize the model architecture and explore diverse user interaction methods to meet broader application needs, emphasizing the importance of understanding user intent and multimodal references [36].

Core Insights - The article discusses the challenges in Multi-Instance Image Generation (MIG), particularly in balancing layout control and identity consistency with reference images [1][3] - A new framework called ContextGen, developed by Zhejiang University's ReLER team, addresses these challenges by utilizing a dual-context attention mechanism [4][52] - ContextGen achieves state-of-the-art (SOTA) performance in various benchmarks, demonstrating significant improvements in spatial accuracy and identity preservation [19][20][24] Group 1: Challenges in MIG - Existing methods struggle to maintain a balance between layout control and identity consistency when generating multiple instances [1][3] - Techniques that allow explicit layout control often fail to customize instances based on reference images [2] - Conversely, methods that utilize reference images struggle with precise layout control and face identity information loss as instance numbers increase [3] Group 2: ContextGen Framework - ContextGen employs a hierarchical decoupling of context to solve the issues of layout control and identity fidelity [5] - The framework introduces a dual-context attention mechanism that integrates global control and local identity injection tasks at different levels of the DiT model [7][52] - Contextual Layout Anchoring (CLA) is used for robust global structure and position anchoring by integrating layout images with instance location information [8][9] - Identity Consistency Attention (ICA) addresses detail loss, particularly in overlapping areas, ensuring high-fidelity identity injection [11][12] Group 3: Data and Optimization - The IMIG-100K dataset, a large-scale synthetic dataset designed for image-guided multi-instance generation tasks, has been released to address the scarcity of high-quality training data [13][14] - ContextGen incorporates a reinforcement learning phase based on preference optimization (DPO) to encourage diverse image generation while maintaining identity [16][19] Group 4: Performance Metrics - ContextGen shows a 5.9% improvement in spatial accuracy (mIoU) on the COCO-MIG benchmark compared to baseline models [20] - In the LayoutSAM-Eval benchmark, ContextGen achieves SOTA across multiple metrics, particularly in maintaining instance attributes such as color, texture, and shape [20][24] - The framework outperforms existing open-source and closed-source models in identity preservation capabilities [24][26] Group 5: User Experience and Future Directions - A user-friendly front-end has been developed to support multiple reference image uploads, automatic image segmentation, and custom layout design [50] - The article emphasizes the importance of dynamic identity adaptation as generative models evolve, highlighting the need for better understanding and coordination of user text intentions and visual references [53]