medium10 pts · first answer countsquantizationllm-serving
Memory for 4-bit weights
Roughly how much memory do the weights of a 7-billion-parameter model need when quantised to 4 bits, ignoring scales and other overhead?
Discussion 0
Join free to post in the discussion.
Related questions
- KV cache size for one sequenceMachine Learning Engineer
- Speculative decoding and output qualityMachine Learning Engineer
- Why serving engines page the KV cacheMachine Learning Engineer
- INT8 activations hurt accuracyMachine Learning Engineer
- Daily cost of an LLM featureMachine Learning Engineer
- Memory for 4-bit weightsAI Engineer (LLM & Generative AI)
- Why long contexts eat GPU memoryAI Engineer (LLM & Generative AI)
- What streaming improvesAI Engineer (LLM & Generative AI)