hard20 pts · first answer countsllm-servingkv-cache
Why serving engines page the KV cache
Why do LLM serving engines store the KV cache in fixed-size blocks (paged attention) instead of one contiguous buffer per request?
Discussion 0
Join free to post in the discussion.
Related questions
- Memory for 4-bit weightsMachine Learning Engineer
- KV cache size for one sequenceMachine Learning Engineer
- Speculative decoding and output qualityMachine Learning Engineer
- Daily cost of an LLM featureMachine Learning Engineer
- Why long contexts eat GPU memoryAI Engineer (LLM & Generative AI)
- What streaming improvesAI Engineer (LLM & Generative AI)
- Sizing the KV cacheAI Engineer (LLM & Generative AI)
- Memory for 7B parameters in fp16MLOps Engineer