Skip to content
RED SAGSkills
hard20 pts · first answer countsmodel-servingcapacity-planninglatency

Latency explodes near full load

At 60% of max throughput, p99 latency is 2 seconds. At 95% it is 25 seconds, although each request's compute time is unchanged. What explains this?

Choose one answer

Discussion 0

Join free to post in the discussion.

Related questions

More Model serving and inference questionsMore Capacity Planning questionsMore Latency questions