Meta详细阐述基于LLM级训练、混合并行计算与知识迁移的GEM广告模型

Core Insights - Meta has released detailed information about its Generative Advertising Model (GEM), aimed at improving ad recommendation capabilities on its platform by processing billions of user-ad interaction data daily [2] - The model addresses the core challenge in recommendation systems, which is the sparsity of meaningful signals such as clicks and conversions [2] - GEM is designed to learn from diverse advertising data, including advertiser goals, creative formats, measurement signals, and user behavior across multiple channels [2] Model Architecture and Training - Meta has redesigned its training architecture to support GEM at a scale comparable to modern large language models, employing customized multi-dimensional parallel strategies for different model components [4] - Dense model components utilize Hybrid Sharded Distributed Parallel (HSDP) technology to optimize memory usage and reduce communication overhead, while sparse components use a two-dimensional parallel scheme combining data and model parallelism [4] - Several GPU-level optimizations have been implemented to reduce training bottlenecks, including custom GPU kernels for variable-length user sequences and memory compression techniques [4] Efficiency and Knowledge Transfer - The system continuously optimizes GPU efficiency throughout the model lifecycle, with lightweight model variants supporting over half of the experiments at a lower cost [5] - Meta employs two migration strategies to transfer the capabilities of the infrastructure model into measurable benefits for user-facing vertical models: direct migration and hierarchical migration [5][6] - These methods maximize transfer efficiency within Meta's advertising model ecosystem through knowledge distillation, representation learning, and parameter sharing [6] Industry Impact and Future Prospects - The effective floating-point operation performance of GEM has improved by 23 times, which is seen as a key factor in changing economic benefits [8] - The technology is viewed as a game changer for advertisers, potentially saving small businesses significant amounts of money by relying on intelligent models to optimize ad spending [9] - Meta envisions that the foundational model for ad recommendation will evolve to better understand user preferences and intentions, facilitating more personalized interactions between users and ads [10]