Continuous batching in LLMs, clearly explained:(a popular LLM interview question; bookmark this)In traditional ML inference, a batch is a matrix.Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.LLM decoding does not work that way.One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.Under static batching, membership is fixed when the ...