hard20 pts · first answer countsmodel-servingoptimization
Why servers capture CUDA graphs
At small batch sizes, an LLM server spends much of each decode step launching many tiny GPU kernels. What does capturing CUDA graphs do?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Latency and throughputMLOps Engineer
- What TTFT measuresMLOps Engineer
- Gap between streamed tokensMLOps Engineer
- Two phases of LLM generationMLOps Engineer