medium10 pts · first answer countsmodel-servinggpu
Low GPU use under load
A GPU model server handles one request at a time. GPU utilisation is 15% while requests queue up. What change helps most?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Scheduling a GPU podMLOps Engineer
- Silent wrong predictions from bad inputMLOps Engineer
- p99 latency spikes during traffic burstsMLOps Engineer
- Why LLM servers use continuous batchingMLOps Engineer