medium10 pts · first answer countsautoscalingkubernetes

Autoscaling a GPU inference service

A GPU inference service autoscales on CPU utilisation. Latency is poor during peaks, but CPU stays at 30%, so it never scales out. What is a better scaling signal?

A.Request queue depth or in-flight requests per replicaB.Disk usageC.Number of log linesD.Memory of the load balancer

← Pod killed while loading the model Concurrent requests at peak →