You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?

What the interviewer wants to hear

• Choose the scheme: weight-only versus weights and activations, 8-bit versus 4-bit
• Use calibration data from the real domain
• Handle outliers with per-channel or group scales or mixed precision
• Evaluate on task metrics, not only perplexity
• Measure real latency, memory and throughput on target hardware

Model answer

I start by setting a quality bar on our own evaluation set and measuring the full-precision model as a baseline. Then I try the least risky option first, usually 8-bit weight-only quantisation, which often has almost no quality loss and halves memory. If I need more savings I try 4-bit weights with group-wise scales using a post-training method that uses calibration data. That data should look like real production inputs. Quantising activations as well gives more speed, but outlier channels can hurt accuracy, so I use per-token or per-channel scales or keep sensitive layers in higher precision. I evaluate on task metrics and edge cases, not just perplexity, since small averages can hide failures. Finally I benchmark latency, throughput and memory on the actual serving hardware, because the speed-up depends on kernel support.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?
• You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?
• How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?
• A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?
• How would you design LLM inference serving to maximise throughput while keeping latency acceptable?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?
• Design a real-time fraud detection system for card payments.