hard20 pts · first answer countslatencydeployment

Where generation time goes

A request has a 6,000-token prompt and an 800-token answer, and total latency is too high. Which statement about prefill and decode is correct?

A.Prefill handles one prompt token at a time, so prompt length dominates total timeB.Decode is compute-bound, so a faster GPU core clock is the main fixC.Both phases cost the same per token, so cut whichever is longerD.Prefill processes the prompt in parallel; decode emits tokens one by one, so the long answer usually dominates

Discussion 0

Join free to post in the discussion.

Where generation time goes

Discussion 0

Related questions