A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?

What the interviewer wants to hear

• Log served features and predictions; replay offline
• Compare feature distributions between training and serving
• Common causes: different code paths, time travel leakage, defaults for missing values
• Prevent with shared feature logic and point-in-time joins
• Add tests and monitoring for skew

Model answer

First I make sure I can see what production sees: log the exact feature vectors and predictions for a sample of requests. Then I recompute the same rows with the offline pipeline and compare feature by feature. Large differences point to the cause. Typical causes are separate code for training and serving, such as SQL versus a Java service, different handling of missing values, unit or timezone differences, and stale caches. Another cause is leakage offline: training features built with data from after the prediction time, which makes offline scores too good. To prevent it, I use one feature definition for both paths, often through a feature store, and build training data from logged features or point-in-time joins. I add unit tests that compare both paths on fixed examples, and monitoring that alerts when serving feature distributions move away from training.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?
• How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?
• A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?
• How would you design LLM inference serving to maximise throughput while keeping latency acceptable?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?
• Design a real-time fraud detection system for card payments.
• You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?