hard20 pts · first answer countsdeploymentinference

Continuous batching in serving

You self-host a model with an inference server that uses continuous batching. What is the main effect compared with static batching?

A.It lowers memory use because it no longer needs a KV cacheB.Requests join the batch as others finish, so GPU throughput risesC.It guarantees each request the same latency as running aloneD.It removes the need to tokenize prompts before the forward pass

Discussion 0

Join free to post in the discussion.

Continuous batching in serving

Discussion 0

Related questions