How do you decide between batch predictions and real-time (online) inference for a model?

What the interviewer wants to hear

• Freshness needed: do predictions depend on very recent data
• Latency and throughput needs
• Cost and operational complexity
• Batch precomputes for known entities; online handles new inputs
• Hybrid approaches are common

Model answer

I start with the product need. If predictions can be a few hours old and the set of entities is known, like daily churn scores for all customers, batch inference is simpler and cheaper: run a scheduled job, write scores to a table and let other systems read them. If the prediction depends on what is happening right now, like fraud checks during payment or search ranking for a new query, I need online inference behind an API with tight latency limits. Online serving adds work: autoscaling, low-latency feature lookups, monitoring and fallbacks. Many systems are hybrid: batch-compute heavy features or candidate lists, then do a light online model at request time. I also consider cost per prediction, peak traffic and what happens if the model service is down, and design a safe fallback.

More medium Machine Learning Engineer questions

• You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
• What problems does a feature store solve, and when is it worth adding one?
• Your neural network loss is not decreasing. How do you debug it?
• After deploying a model, what would you monitor, and how do you handle data drift?
• Walk me through how you would build a RAG system to answer questions over a company knowledge base.
• How would you evaluate an LLM-powered feature before and after launch?
• How do you approach hyperparameter tuning for a model on a limited compute budget?
• A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
• Your new model scores far higher than anything before it. How do you check for data leakage?