hard20 pts · first answer countsllm-servinggpu-memory
KV cache with grouped-query attention
A model has 32 query heads but uses grouped-query attention with 8 key/value heads. Compared with standard multi-head attention, how big is its KV cache?
Discussion 0
Join free to post in the discussion.
Related questions
- Memory for 7B parameters in fp16MLOps Engineer
- KV cache size for one long requestMLOps Engineer
- Serving many fine-tuned variantsMLOps Engineer
- Why LLM servers use continuous batchingMLOps Engineer
- Where to use tensor parallelismMLOps Engineer
- Memory to fully fine-tune 7B with AdamMLOps Engineer
- Semantic cache returns wrong answersMLOps Engineer
- Memory for 4-bit weightsMachine Learning Engineer