Skip to content
RED SAGSkills
medium10 pts · first answer countsmodel-servingperformancetooling

Small model leaves the GPU idle

A small BERT model in Triton uses only 25% of a GPU even under load, because each request does a short run with gaps between. What simple change can raise use?

Choose one answer

Discussion 0

Join free to post in the discussion.

Related questions

More Model serving and inference questionsMore Performance and debugging questionsMore Tooling questions