hard20 pts · first answer countslatencyllm-serving

What streaming improves

You enable token streaming in a chat app. Users say it "feels much faster". Which statement is correct?

A.Streaming reduces the total tokens generated, lowering costB.Streaming makes the model think faster per tokenC.Streaming mainly improves perceived latency by showing tokens as they arrive (time to first token); total generation time is about the sameD.Streaming removes the need for a context window

Discussion 0

Join free to post in the discussion.

What streaming improves

Discussion 0

Related questions