easy5 pts · first answer countstransformersinference

Why output is slower than input

A 3,000-token prompt is processed in about a second, but a 300-token answer takes several seconds. Why?

A.The output uses a larger model than the one that reads the promptB.Prompt tokens are processed in parallel; output tokens one after anotherC.Output tokens are checked by a human reviewer before sendingD.The prompt is cached, so it takes no compute at all

Discussion 0

Join free to post in the discussion.

Why output is slower than input

Discussion 0

Related questions