hard20 pts · first answer countsmodel-servingperformance
When decode stops being memory-bound
A GPU offers about 990 TFLOPS of FP16 compute and 3.35 TB/s bandwidth. FP16 decode does about 2 FLOPs per weight per sequence and reads 2 bytes per weight per step. Roughly what batch size makes decode compute-bound?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Latency and throughputMLOps Engineer
- What TTFT measuresMLOps Engineer
- Gap between streamed tokensMLOps Engineer
- Two phases of LLM generationMLOps Engineer