hard20 pts · first answer countsautoscalingmodel-serving

p99 latency spikes during traffic bursts

A GPU model service autoscales from 2 to 10 replicas. During morning bursts, p99 latency jumps to 40 seconds for a few minutes. New pods take time to pull a 15 GB image and load weights. What best addresses this?

A.Remove autoscaling and use only 2 replicasB.Reduce cold start (pre-pulled or smaller images, cached weights on fast storage), keep warm minimum capacity, and scale earlier on queue depth or a scheduleC.Increase the request timeout to 60 secondsD.Switch the health check to always return success

Discussion 0

Join free to post in the discussion.

p99 latency spikes during traffic bursts

Discussion 0

Related questions