medium10 pts · first answer countstransformersinference

What limits prefill and decode speed

In LLM serving, which statement about prefill and decode is generally true at small batch sizes?

A.Prefill is memory-bound; decode is compute-boundB.Both are limited only by network speedC.Both are limited only by the tokenizer's speedD.Prefill is compute-bound; decode is memory-bandwidth-bound

Discussion 0

Join free to post in the discussion.

What limits prefill and decode speed

Discussion 0

Related questions