medium10 pts · first answer countsquantizationdeployment
Memory for 4-bit weights
Roughly how much memory do the weights of an 8-billion-parameter model need when quantised to 4 bits per parameter (ignoring small overheads)?
Discussion 0
Join free to post in the discussion.
Related questions
- Sizing the KV cacheAI Engineer (LLM & Generative AI)
- Where generation time goesAI Engineer (LLM & Generative AI)
- Continuous batching in servingAI Engineer (LLM & Generative AI)
- Serving a mixture-of-experts modelAI Engineer (LLM & Generative AI)
- Errors during every deployBackend Developer
- Liveness probe restarts every podBackend Developer
- Adding an index to a live big tableBackend Developer
- Docker image vs containerFull Stack Developer