hard20 pts · first answer countskv-cachellm-serving
KV cache size for one sequence
A decoder has 32 layers and hidden size 4096 with standard multi-head attention (no grouped-query sharing). In fp16, how much KV cache does one 4,096-token sequence need?
Discussion 0
Join free to post in the discussion.
Related questions
- Memory for 4-bit weightsMachine Learning Engineer
- Speculative decoding and output qualityMachine Learning Engineer
- Why serving engines page the KV cacheMachine Learning Engineer
- Daily cost of an LLM featureMachine Learning Engineer
- Why long contexts eat GPU memoryAI Engineer (LLM & Generative AI)
- What streaming improvesAI Engineer (LLM & Generative AI)
- Sizing the KV cacheAI Engineer (LLM & Generative AI)
- Memory for 7B parameters in fp16MLOps Engineer