hard20 pts · first answer countstransformersinference

GPU at low utilisation during decoding

During single-user decoding, GPU compute utilisation is about 5%, yet tokens/s will not go higher. Adding more concurrent users barely slows each one at first. What explains this?

A.The GPU is throttling its clock because it is too hotB.The tokenizer runs on the GPU and blocks the modelC.Decode waits on weight reads, which extra sequences shareD.Each user is served by a separate idle GPU core

Discussion 0

Join free to post in the discussion.

GPU at low utilisation during decoding

Discussion 0

Related questions