So joining us now is Austin Lions, senior analyst at Creative Strategies with a closer look at some of the latest tech developments. And so Austin, thank you so much for joining us on this Friday. Always appreciate your time and all of your thoughts.And you have said that the GPUs for everything era is maybe over. And so what does that mean for say like Nvidia's mode as more and more of these AIS players like rock and cerebras tried to slot into inference. >> Sure.Yes. So, historically, Nvidia was the first ...

RT Avi Chawla (@_avichawla)8x faster LLM inference than Cerebras is here!!And it generates 17,000 tokens per second.Today, a key bottleneck in how LLM inference works is that when you run a model on any GPU, the model weights live in memory, and the compute cores have to constantly fetch those weights to do math.That back-and-forth between memory and compute is the single biggest reason inference is slow. It's also the reason we need expensive HBM stacks, liquid cooling, and high-speed interconnects, making ...

If you found it insightful, reshare it with your network.Find me → @_avichawlaEvery day, I share tutorials and insights on DS, ML, LLMs, and RAGs. https://t.co/OOzQAIhOFjAvi Chawla (@_avichawla):8x faster LLM inference than Cerebras is here!!And it generates 17,000 tokens per second.Today, a key bottleneck in how LLM inference works is that when you run a model on any GPU, the model weights live in memory, and the compute cores have to constantly fetch those weights https://t.co/9el7M7Dlsm ...

8x faster LLM inference than Cerebras is here!!And it generates 17,000 tokens per second.Today, a key bottleneck in how LLM inference works is that when you run a model on any GPU, the model weights live in memory, and the compute cores have to constantly fetch those weights to do math.That back-and-forth between memory and compute is the single biggest reason inference is slow. It's also the reason we need expensive HBM stacks, liquid cooling, and high-speed interconnects, making AI data centers costly.Taa ...

Learn how LLM inference actually works under the hood.vLLM has 100k+ lines of code. Mini-SGLang does the same core job in 5,000.It's a compact codebase that serves as both a capable inference engine and a transparent reference for researchers and devs. Something you can actually finish reading over a weekend.Here's what makes it special:↳ Clean, type-annotated code you can actually read↳ Radix cache to reuse KV cache across shared prefixes↳ Chunked prefill for long contexts without memory blowup↳ Tensor par ...

Zero-Knowledge Proofs (ZKP) Application - ZKP allows service providers to prove the correctness of LLM inference without revealing model parameters [1] - ZKP can address the issue of service providers potentially deploying smaller/cheaper models than promised [1] zkGPT Overview - The report introduces new work on zkGPT, focusing on proving LLM inference fast with Zero-Knowledge Proofs [1]

Core Technique - KV caching is a technique used to speed up LLM inference [1] Explanation Resource - Avi Chawla provides a clear explanation of KV caching in LLMs with visuals [1]