hard20 pts · first answer countsmodel-servingcapacity-planning
Throughput versus goodput
Raising max batch size lifts throughput from 4,000 to 6,000 tokens per second, but the share of requests meeting the TTFT SLO falls from 98% to 60%. Which metric shows this trade-off best?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Throughput of four replicasMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Securing GPUs for a 3-month runMLOps Engineer
- Latency and throughputMLOps Engineer
- What TTFT measuresMLOps Engineer