medium10 pts · first answer countsllm-servinggpu-memory
KV cache size for one long request
A model has 32 layers and hidden size 4096, uses standard multi-head attention and stores the KV cache in fp16. Roughly how much KV cache memory does one 4,096-token sequence need?
Discussion 0
Join free to post in the discussion.
Related questions
- Memory for 7B parameters in fp16MLOps Engineer
- Serving many fine-tuned variantsMLOps Engineer
- Why LLM servers use continuous batchingMLOps Engineer
- Where to use tensor parallelismMLOps Engineer
- KV cache with grouped-query attentionMLOps Engineer
- Memory to fully fine-tune 7B with AdamMLOps Engineer
- Semantic cache returns wrong answersMLOps Engineer
- Memory for 4-bit weightsMachine Learning Engineer