How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?

What the interviewer wants to hear

• Define a primary metric and guardrail metrics in advance
• Power analysis for sample size and duration
• Randomise by user; avoid peeking
• Novelty effects, network effects, feedback loops
• Shadow or canary first for safety

Model answer

Before the test I run the new model in shadow mode to check latency, errors and prediction distributions without affecting users, then a small canary. For the A/B test, I define one primary metric, like conversion rate, and guardrail metrics such as latency, revenue per user and complaint rate. I do a power analysis to pick the sample size and run for at least one or two full weekly cycles. I randomise by user, not by request, so each person has a consistent experience. Pitfalls include peeking and stopping early when results look good, which inflates false positives; novelty effects that fade; interference when users affect each other, like marketplaces; and feedback loops where the model changes the data it later trains on. I also check sample ratio mismatch and segment results, and only ship when the gain is significant and guardrails hold.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?
• You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?
• A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?
• How would you design LLM inference serving to maximise throughput while keeping latency acceptable?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?
• Design a real-time fraud detection system for card payments.
• You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?