hard20 pts · first answer countsmodel-servingllm-inferenceperformance
Upper bound on single-stream speed
An 8B model in FP16 has 16 GB of weights. On a GPU with 3.35 TB/s memory bandwidth, what is the rough upper limit on decode speed for a single sequence (batch size 1)?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Latency and throughputMLOps Engineer
- What TTFT measuresMLOps Engineer
- Gap between streamed tokensMLOps Engineer
- Two phases of LLM generationMLOps Engineer