hard20 pts · first answer countsllm-servingtransformers
Why long contexts eat GPU memory
When serving a self-hosted LLM, GPU memory runs out as users send longer prompts and you increase concurrency. What mainly grows with sequence length and batch size?
Discussion 0
Join free to post in the discussion.
Related questions
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- What streaming improvesAI Engineer (LLM & Generative AI)
- Sizing the KV cacheAI Engineer (LLM & Generative AI)
- Why attention divides by sqrt(d_k)AI Engineer (LLM & Generative AI)
- What the causal mask preventsAI Engineer (LLM & Generative AI)
- Scaling factor in attentionMachine Learning Engineer
- Memory for 4-bit weightsMachine Learning Engineer
- KV cache size for one sequenceMachine Learning Engineer