Thank you, everyone, for joining this session. I'm very excited to talk about the SGLN Miles, two open source frameworks that we have spent time building on this that aim to help you build frontier AI infrastructure. Let's start with, I mean, what is SGLN.Some of you might already heard of and maybe are using it. It's an open source framework for inference and has been widely adopted in production. People use it to serve open source models in many companies across all the categories.It aims for a target for ...

Continuous batching in LLMs, clearly explained:(a popular LLM interview question; bookmark this)In traditional ML inference, a batch is a matrix.Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.LLM decoding does not work that way.One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.Under static batching, membership is fixed when the ...