hard20 pts · first answer countsllm-servinggpu

Why LLM servers use continuous batching

Why do modern LLM serving engines use continuous (in-flight) batching instead of static batching?

A.Requests have very different output lengths; continuous batching adds and removes sequences at each generation step, so the GPU is not idle waiting for the longest request in a batchB.It removes the need for a KV cacheC.It makes each token cheaper to compute on CPUD.It guarantees the same output for every request

Discussion 0

Join free to post in the discussion.

Why LLM servers use continuous batching

Discussion 0

Related questions