How do you decide between batch predictions and a real-time model API?

What the interviewer wants to hear

• Depends on when the prediction is needed and how fresh the inputs must be
• Batch is cheaper and simpler for scheduled use such as daily scores
• Real-time is needed for decisions during a user request
• Consider latency targets, cost and operational effort

Model answer

I start with when the prediction is needed and how fresh the inputs must be. If the business uses predictions on a schedule, like a daily churn list for the retention team or weekly demand forecasts, batch scoring is simpler and cheaper. It runs on a schedule, writes results to a table and is easy to rerun if something fails. If the decision happens inside a user request, like fraud checks at checkout or search ranking, I need a real-time API with a latency target, autoscaling and on-call support. There is also a middle option: precompute scores in batch and look them up in real time, or use streaming features when only part of the input must be fresh. I pick the simplest option that meets the freshness and latency need, because real-time systems cost more to build and run.

More easy MLOps Engineer questions

• What is MLOps, and how is it different from traditional DevOps?
• Why does a team need a model registry? What information should it store?
• How do you make a model training run reproducible?
• How would you deploy a trained scikit-learn model as a REST API?
• What would you monitor for a model running in production?
• What is training-serving skew, and how do you prevent it?
• Explain the difference between data drift and concept drift, and how you would detect each.
• What makes a good Docker image for a model-serving service?
• Which Kubernetes concepts matter most when you deploy a model service, and why?