Core Viewpoint - EXO Labs has developed a new framework that enhances large model inference speed by combining NVIDIA's DGX Spark with Apple's M3 Ultra, achieving a speedup of up to 2.77 times for model deployment [1][5][18]. Group 1: Technology and Implementation - The framework utilizes a PD (Prefill and Decode) separation approach, where DGX Spark handles the Prefill phase due to its high computational power, while M3 Ultra manages the Decode phase, benefiting from its high memory bandwidth [11][18]. - The Prefill phase's computational demand grows quadratically with prompt length, while the Decode phase is primarily limited by memory bandwidth, making the separation of tasks advantageous [8][11]. - EXO Labs employs a streaming transmission method for KV cache, allowing for overlapping computation and data transfer between the two devices, which minimizes communication costs [16][18]. Group 2: Performance Metrics - The combination of DGX Spark and M3 Ultra results in significant performance improvements: Prefill speed increases to 3.79 times that of M3 Ultra alone, and Decode speed improves to 3.37 times that of DGX Spark [18][19]. - The overall performance metrics show that the combined system reduces total processing time to 2.32 seconds, achieving a speedup of 2.8 times compared to using M3 Ultra alone [19]. Group 3: Industry Context - NVIDIA is also exploring similar PD separation techniques with its upcoming Rubin CPX platform, which will utilize a compute-intensive processor for Prefill and a high-bandwidth memory chip for Decode [20]. - The recent delivery of DGX Spark systems to notable figures in the tech industry indicates a growing interest and investment in advanced AI inference technologies [22]. - Apple's latest M5 chip shows improvements in AI performance, but comparisons suggest that M3 Ultra may hold more value in the current landscape of AI hardware [26][30].