easy5 pts · first answer countsgpu-memoryllm-serving
Memory for 7B parameters in fp16
Roughly how much GPU memory do the weights of a 7-billion-parameter model need in 16-bit precision?
Discussion 0
Join free to post in the discussion.
Related questions
- KV cache size for one long requestMLOps Engineer
- Serving many fine-tuned variantsMLOps Engineer
- Why LLM servers use continuous batchingMLOps Engineer
- Where to use tensor parallelismMLOps Engineer
- KV cache with grouped-query attentionMLOps Engineer
- Memory to fully fine-tune 7B with AdamMLOps Engineer
- Semantic cache returns wrong answersMLOps Engineer
- Memory for 4-bit weightsMachine Learning Engineer