MLOps Engineer interview questions and answers

MLOps is a set of practices and tools for taking machine learning models into production and keeping them working. It builds on DevOps ideas like automation, CI/CD, infrastructure as code and monitoring. The difference is that an ML system has three moving parts, not one: code, data and the trained model. A model can break even when the code has not changed, because the incoming data changes. So MLOps adds experiment tracking to compare runs, data versioning for reproducibility, a model registry to manage versions and approvals, and pipelines that retrain and evaluate models automatically. Testing also covers data quality and model quality, not just unit tests. Finally, monitoring includes drift and prediction quality, not only uptime and latency, with a process to retrain when performance falls.

Without a registry, models end up as files in random buckets and nobody is sure which one is in production or how it was built. A model registry is the single source of truth. For every version it stores the artifact plus metadata: evaluation metrics, the training run, code commit, data version, dependencies, the owner and a description. It also tracks lifecycle state, such as candidate, staging, production or archived, often with approval steps. Deployment pipelines pull a specific registered version rather than a loose file, so every prediction service can be traced back to its training data and code. That makes audits easy and rollback simple: point production back to the previous approved version. Tools like MLflow provide this, and cloud platforms have their own registries.

I make sure every input to training is recorded and fixed. Code is in Git and every run logs its commit. Data is versioned, either as snapshots with a tool like DVC, as table versions in a lakehouse, or as a query with a fixed time range, and the version is logged. Hyperparameters live in a config file, not hard-coded. The environment is fixed with a lock file and a Docker image, so library versions do not change silently. I set random seeds for Python, NumPy and the ML framework, and note the hardware type, since GPUs can still introduce tiny differences. An experiment tracker such as MLflow records all of this along with metrics and the output model. With that, anyone on the team can rerun a past experiment and get the same or nearly the same result.

I save the trained model, ideally as a full pipeline including preprocessing, and register it. Then I write a small service with a framework like FastAPI that loads the model once at startup, not on every request. A predict endpoint accepts JSON, validates it with a schema so wrong types or missing fields return a clear error, runs the model and returns predictions. I add a health endpoint for readiness and liveness checks, structured logging of inputs and outputs (with sensitive data removed), and metrics like request count, latency and errors. I package it in a Docker image with pinned versions, the same ones used during training. Finally I deploy it to Kubernetes or a managed container service with autoscaling, run a load test, and add monitoring dashboards and alerts.

I monitor four things. First, service health, like any API: request rate, latency at p50, p95 and p99, error rate, and CPU, memory or GPU use. Second, data quality: schema changes, missing values, values outside expected ranges and new categories. These often catch upstream bugs early. Third, drift: compare feature distributions and the prediction distribution with the training data or a recent baseline. A sudden jump in the share of positive predictions is a strong warning. Fourth, model and business performance: when true labels arrive, compute metrics like precision or error over time and by segment, and track the business KPI the model supports. All of this goes to dashboards with alerts that have sensible thresholds and owners, so the right person acts when something goes wrong.

Training-serving skew happens when the features a model receives in production are not computed the same way as the features it was trained on. A common cause is writing the logic twice, for example in SQL for training and in application code for serving, and the two drift apart in rounding, time zones or how missing values are filled. It can also come from using different data sources or from timing, such as using data in training that is not yet available at request time. To prevent it, I keep a single feature definition, either in a feature store or by packaging the preprocessing inside the model artifact, so both paths run the same code. I also log the features actually used at serving time and run a regular parity check that compares them with what the offline pipeline would produce for the same entities.

I start with when the prediction is needed and how fresh the inputs must be. If the business uses predictions on a schedule, like a daily churn list for the retention team or weekly demand forecasts, batch scoring is simpler and cheaper. It runs on a schedule, writes results to a table and is easy to rerun if something fails. If the decision happens inside a user request, like fraud checks at checkout or search ranking, I need a real-time API with a latency target, autoscaling and on-call support. There is also a middle option: precompute scores in batch and look them up in real time, or use streaming features when only part of the input must be fresh. I pick the simplest option that meets the freshness and latency need, because real-time systems cost more to build and run.

Data drift means the inputs change: for example, a new marketing campaign brings in younger users, so the age distribution shifts. Concept drift means the relationship between inputs and the target changes: the same transaction pattern that used to be safe is now fraud because fraudsters adapted. Data drift can be detected without labels by comparing live feature and prediction distributions with the training data, using PSI, KS tests or a classifier that tries to tell the two apart. Concept drift is harder because inputs may look normal. It usually shows up as falling performance once labels arrive, so I track metrics over time on delayed labels and use proxy signals such as complaint rates in the meantime. In both cases I alert on meaningful changes in important features or model performance, not on every small statistical difference.

I start from a small, pinned base image, such as a slim Python image with a specific version, and install pinned dependencies from a lock file. I copy the requirements file and install packages before copying the application code, so code changes do not reinstall everything. If packages need compilers, I use a multi-stage build so the final image only contains what runs. I never bake secrets or training data into the image; secrets are injected at runtime. The container runs as a non-root user and exposes health and readiness endpoints. For the model itself, I either download a specific version from the registry at startup or include it in the image and record the version, so every running container can be traced to an exact model. Finally I scan the image for vulnerabilities in CI and keep it as small as practical so it starts quickly.

The core object is a Deployment, which runs a set number of identical pods and handles rolling updates, so a new model version can replace the old one gradually. A Service gives those pods a stable address and load-balances between them. Resource requests and limits matter a lot for models: memory requests must cover model loading, and GPU workloads request GPUs explicitly so they land on the right nodes. Readiness probes stop traffic reaching a pod until the model is loaded, and liveness probes restart pods that hang, but with a long enough start delay. ConfigMaps and Secrets keep configuration and credentials out of the image. Finally, a Horizontal Pod Autoscaler adds replicas under load, ideally scaling on request rate or queue depth rather than CPU for GPU services.

On each pull request, CI runs unit tests for feature code and the serving app, linting, and data validation checks on schemas and expectations. A quick smoke test trains on a small sample to make sure the pipeline runs end to end. When changes merge, the training pipeline runs on full data and registers a candidate model with its metrics. An evaluation gate compares it with the current production model on the same holdout set, and also checks slices, fairness and latency. If it passes, CD builds the serving image, deploys it to staging, runs integration and load tests, then rolls out with shadow or canary traffic while watching metrics. Bad signals trigger automatic rollback. For high-risk models I add a human approval step in the registry before production.

In a shadow deployment, the new model receives a copy of live requests but its outputs are only logged, so users are not affected. I use it to check latency, errors and how predictions differ from the current model on real data. In a canary, a small share of real traffic, say 5%, is served by the new model; if health and key metrics stay good, I ramp up gradually. It is mainly a safety check. An A/B test splits users randomly for long enough to measure business impact, like conversion, with statistical confidence. I often use all three in order: shadow to catch technical issues, canary for safe exposure, and A/B to prove value. Each stage needs clear success and rollback criteria decided in advance, and automation to roll back fast.

I would use an orchestrator like Airflow or Kubeflow Pipelines. A weekly schedule starts the pipeline, and drift or performance alerts can also trigger it. Step one extracts a fixed time window of data and runs validation: schema, row counts, missing values and label distribution. If validation fails, the pipeline stops and alerts, since training on broken data is worse than not training. Step two trains with a versioned config and logs everything to the experiment tracker. Step three evaluates the new model against the current production model on a recent holdout, overall and by important segments, plus latency checks. If it wins by a set margin and passes all gates, it is registered and promoted, then deployed by canary. Every step is idempotent so retries are safe, and failures notify the owning team.

The biggest challenge is consistency between offline and online stores. I make both come from the same feature definitions and materialisation jobs, and run regular checks that compare samples from both. Freshness is next: each feature needs an agreed update frequency and alerts when materialisation jobs are late. Backfills are needed when a feature changes or is added, and they must be point-in-time correct so training does not leak future data. The online store must be fast and highly available because every prediction depends on it, so I set latency targets, replicate it, and have fallbacks or default values if a lookup fails. Finally, governance: every feature needs an owner and documentation, and I track usage so unused features can be removed instead of costing compute forever.

I set up dedicated GPU node pools with the NVIDIA device plugin, and taint those nodes so only pods that tolerate the taint and request nvidia.com/gpu land there. That stops ordinary workloads from wasting expensive nodes. For the service itself, I use a proper inference server, like Triton or vLLM for LLMs, with dynamic or continuous batching to keep the GPU busy. Autoscaling on CPU usage does not work well for GPU services, so I scale on queue length, concurrency or GPU utilisation through custom metrics. Because GPU pods start slowly, I reduce image size, cache model weights near the node, and keep a minimum number of warm replicas. For small models, GPU sharing such as MIG or time slicing improves use. I monitor utilisation and cost per thousand predictions to keep spend under control.

I split the pipeline into small tasks such as extract, validate, build features, train, evaluate and register, each with clear inputs and outputs passed by reference to versioned storage. Every task is idempotent: it writes to a partition keyed by the run date and overwrites it, so rerunning never duplicates data. Tasks have retries with backoff for transient errors and timeouts so a hung job does not block the DAG. Between steps I add gates: data validation before training and a champion comparison before registration, so bad data or a worse model stops the pipeline. For backfills I keep catchup off by default, run them deliberately with a limited number of active runs, and make sure the source systems can handle the load. Finally, each run logs lineage, such as data versions, code commit and resulting model version, so any output can be traced.

I start with weight memory: parameters times bytes per parameter, so a 70B model in 16-bit needs about 140 GB, and 4-bit cuts that to roughly 35 GB. Then I estimate the KV cache: 2 times layers times KV heads times head size times bytes, per token, multiplied by the typical context length and the number of sequences in flight. With grouped-query attention this can be several times smaller than the naive estimate. I add headroom for activations and runtime overhead, usually 10 to 20 percent. That tells me how many GPUs are needed just to fit the model and the concurrency, and whether tensor parallelism is needed. Then I benchmark with a realistic mix of prompt and output lengths to measure tokens per second and time to first token, and size the fleet for peak traffic with margin.

I start from what users of the model need. For an online service the basic indicators are availability, measured as successful responses, and latency, usually p95 or p99 against a target such as 150 ms. For ML I add indicators that normal services lack: freshness of features and of batch predictions, for example scores updated by 6 am every day, and quality signals such as prediction distribution stability and accuracy on delayed labels. Each SLO has a target and an error budget, such as 99.9% availability over 30 days, which is about 43 minutes of downtime. If we burn the budget quickly, we slow down risky changes like model swaps. Alerts fire on budget burn rate and user-facing symptoms rather than on every low-level metric, so on-call engineers are not flooded with noise.

I treat models, datasets and packages as code that can be attacked. Third-party models come from approved sources and in safe formats like safetensors rather than pickle files, which can run code when loaded. Artifacts in the registry have hashes recorded and, ideally, signatures that the deployment checks. Docker images and Python dependencies are pinned and scanned, and deployments use image digests rather than mutable tags. Pipelines and serving jobs run with least-privilege service accounts, so a training job cannot write to production, and secrets come from a secrets manager. Training data access is logged, and sensitive data is masked where possible. Finally I keep an audit trail linking each deployed model to its code commit, data version, base model and approver, which helps both investigations and compliance.

On top of uptime and error rates, I trace every request end to end: prompt template version, model, retrieved documents, tool calls, and the latency of each step. I track token usage and cost per feature and per tenant, because costs can jump when prompts or context sizes change. For latency I separate time to first token from total generation time, since streaming makes the first one most visible to users. Quality is harder, so I combine user feedback such as thumbs up and down, automated evaluations on a sample of production traffic using rubric-based judges checked against human labels, and rates of refusals, empty answers and failed JSON parsing. For safety I monitor guardrail triggers, suspected prompt injection and personal data in outputs. All of this is broken down by prompt and model version, so regressions after a change are easy to spot.

I would start by interviewing teams to see their common workflows and pain points, then build a paved road for the most common cases rather than trying to support everything. The core pieces are: governed data access, experiment tracking, a central model registry, a pipeline orchestrator for training and batch jobs, a standard serving layer on Kubernetes for real-time models, a feature store where several teams share features, and built-in monitoring for latency, drift and quality. Teams get self-service templates, such as a project template with CI/CD, Docker and monitoring already wired, so a new model can reach staging in a day. Security and governance are built in: role-based access, secrets handling, lineage and audit logs. I show cost per team. I would migrate one or two willing teams first, measure time to deploy and incidents, then expand.

I would use a dedicated serving engine such as vLLM or a similar runtime, because continuous batching and paged KV cache management give far higher throughput than a simple web server. First I plan GPU memory: the weights plus enough KV cache for expected context lengths and concurrency. Quantisation to 8-bit or 4-bit can cut memory and cost, but I check quality on an eval set. If the model does not fit on one GPU, I use tensor parallelism within a node. I put a gateway in front for authentication, rate limits per tenant, request routing and streaming responses. Autoscaling uses queue depth and GPU metrics, with warm replicas because loading weights is slow. Prefix caching helps when many requests share a system prompt. I monitor time to first token, tokens per second, error rate and cost per million tokens.

First I confirm the drop is real and not a tracking problem, and estimate the impact. I open an incident, tell stakeholders, and if impact is high I mitigate fast: roll back to the previous model version, or switch to a safe fallback like popularity-based recommendations, while investigating. Next I check what changed: model deployments, feature pipeline releases, upstream data sources, config flags and experiments. Most incidents are data problems, so I compare live feature distributions and the prediction distribution with last week. For example, a feature might suddenly be all null because an upstream table was renamed, and the model silently uses default values. Once the root cause is found, I fix it, backfill if needed, and redeploy carefully. Afterwards I write a blameless postmortem and add prevention, such as schema checks and alerts on feature null rates.

I start with visibility: tag every resource by team, project and model, and build a cost dashboard. Usually a few items explain most of the spend. For training, I move fault-tolerant jobs to spot or preemptible instances with regular checkpointing, right-size instance types, and stop idle notebooks and development GPUs automatically. For serving, I look at utilisation: many GPU services run at 10 to 20 percent. Batching, better autoscaling, sharing GPUs between small models, moving small models to CPU, and scale to zero for rarely used endpoints all help. Quantisation or distillation can cut the hardware needed. I also clean up old artifacts, logs and unused features. To keep teams fast, I use budgets, quotas and alerts with clear ownership instead of manual approvals, and share the savings each team achieved.

The platform should make this a lookup, not an investigation. Every model version in the registry links to the exact training data snapshot, feature definitions, code commit, configuration, environment and evaluation results, and these artifacts are immutable. Each version also has documentation such as a model card, validation report, fairness checks and the names of people who approved it, with timestamps. For individual decisions, the serving layer logs a request ID, the model version, the input features used and the output, plus explanation values if required, stored securely with retention rules that match the regulation. Access to these logs is controlled and itself audited. With this, I can answer which model and data produced a decision on a given date, reproduce the prediction, and show the approval trail.

The core is checkpointing. I save complete state, meaning model and optimiser shards, scheduler, step, random states and data loader position, to object storage on a fixed interval, chosen so the expected lost work per preemption is small compared with checkpoint cost. Asynchronous or sharded checkpointing keeps saves from stalling training. I listen for preemption notices and trigger a final save when one arrives. The orchestrator restarts the job automatically on new nodes and resumes from the latest valid checkpoint, verifying it is complete before using it. If node counts can change, I use elastic training or keep the global batch size constant through gradient accumulation. I monitor for stragglers and bad nodes that slow every step. Finally I measure goodput, the share of GPU time that produced progress, so I can compare the spot savings against time lost to restarts.

Vectors from different embedding models live in different spaces, so I cannot mix them; the whole corpus must be re-embedded. I would create a new index alongside the old one and start dual writes, so new and updated documents go to both. Then I run a backfill job over the 200 million chunks, batched for GPU throughput, idempotent and resumable, with rate limits so it does not hurt production. I estimate the cost first, since this can be large. Before switching, I evaluate retrieval on a labelled query set, comparing recall at k and answer quality, and tune index settings for the new dimension. I then shadow or A/B test the new index on live queries. The switch itself is a config flag or index alias, so rollback is instant, and I keep the old index until the new one has proven itself.

First I measure where the time goes. Time to first token is mostly prefill and queueing, while total time is driven by decode speed and output length. If prompts are long, I trim them, cut retrieved context and enable prefix caching so shared system prompts are not recomputed. If queueing dominates, I add capacity or lower the maximum batch size, because large batches raise throughput but slow each request. For decode speed, I try quantised weights, speculative decoding with a small draft model, and tensor parallelism across more GPUs, which lowers per-token latency at some cost. I also look at product choices: shorter outputs, streaming tokens to the user, and routing simple requests to a smaller model. Every change goes through the same quality evaluation set and a load test, so I can see the latency, throughput and quality trade-off before shipping.

The hard part is finding every copy, so the platform needs lineage and a data catalogue that link a user identifier to raw tables, derived features, embeddings in vector indexes, caches, logs and training snapshots. A deletion workflow then removes or anonymises records in each system and records proof for the audit. For online stores and vector indexes this is a direct delete. For training data, snapshots are rebuilt without the user, and for trained models we follow a documented policy agreed with legal: often the next scheduled retrain on cleaned data is enough, but high-risk cases may need retraining sooner. Prevention matters even more: keep personal data out of features and prompts where possible, pseudonymise identifiers, set retention limits on logs and snapshots, and avoid copying data into ad hoc notebooks. The workflow must finish within the legal deadline and be tested regularly.

I start with one feature definition, for example "purchases in the last 24 hours", declared once in a feature store or shared library and compiled to both streaming and batch jobs. The stream job uses event time, not processing time, with watermarks so late events land in the right window. Because most brokers deliver at least once, updates are deduplicated by event ID or written as idempotent upserts. For training, I either log the exact feature values served with each prediction or use point-in-time joins on the batch backfill, so training sees what production would have seen. Then I run continuous parity checks: sample entities, compute the feature both ways and alert when they differ beyond a small tolerance. I also monitor freshness and lag, because a slow stream silently turns real-time features into stale ones.