hard20 pts · first answer countskv-cachetransformersdeployment
Sizing the KV cache
A model has 32 layers, 32 query heads but only 8 key-value heads (grouped-query attention), head dimension 128, and stores the KV cache in 16-bit precision. How much KV cache memory does one 8,192-token sequence need?
Discussion 0
Join free to post in the discussion.
Related questions
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- Memory for 4-bit weightsAI Engineer (LLM & Generative AI)
- Why long contexts eat GPU memoryAI Engineer (LLM & Generative AI)
- Why attention divides by sqrt(d_k)AI Engineer (LLM & Generative AI)
- What the causal mask preventsAI Engineer (LLM & Generative AI)
- Where generation time goesAI Engineer (LLM & Generative AI)
- Continuous batching in servingAI Engineer (LLM & Generative AI)
- Serving a mixture-of-experts modelAI Engineer (LLM & Generative AI)