medium10 pts · first answer countscapacity-planninglatency
Concurrent requests at peak
A model API receives 200 requests per second and each takes 250 ms on average. On average, how many requests are in progress at once?
Discussion 0
Join free to post in the discussion.
Related questions
- What p99 latency meansMLOps Engineer
- Throughput of four replicasMLOps Engineer
- Fleet-wide p99 from many podsMLOps Engineer
- Cost per 1,000 predictionsMLOps Engineer
- Tail latency at low CPU usageMLOps Engineer
- Percentiles vs average latencySite Reliability Engineer
- p99 latency across all instancesSite Reliability Engineer
- A pod stuck in PendingSite Reliability Engineer