How do you build a real-time feature pipeline whose values match the batch features used in training?

What the interviewer wants to hear

• Single feature definition shared by stream and batch
• Event-time processing with watermarks for late data
• Idempotent, deduplicated updates under at-least-once delivery
• Point-in-time correct training data from logged or recomputed features
• Continuous parity checks between online and offline values

Model answer

I start with one feature definition, for example "purchases in the last 24 hours", declared once in a feature store or shared library and compiled to both streaming and batch jobs. The stream job uses event time, not processing time, with watermarks so late events land in the right window. Because most brokers deliver at least once, updates are deduplicated by event ID or written as idempotent upserts. For training, I either log the exact feature values served with each prediction or use point-in-time joins on the batch backfill, so training sees what production would have seen. Then I run continuous parity checks: sample entities, compute the feature both ways and alert when they differ beyond a small tolerance. I also monitor freshness and lag, because a slow stream silently turns real-time features into stale ones.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• How would you self-host and serve an open-weight LLM for thousands of concurrent users?
• Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.
• The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?
• Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?
• Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?