What does a good CI/CD pipeline for machine learning look like?

What the interviewer wants to hear

• CI: unit tests, data validation, linting, small training smoke test
• Training pipeline produces candidate model with metrics
• Automated evaluation gates against current production model
• CD: build image, deploy to staging, canary or shadow rollout
• Automated rollback and approvals where needed

Model answer

On each pull request, CI runs unit tests for feature code and the serving app, linting, and data validation checks on schemas and expectations. A quick smoke test trains on a small sample to make sure the pipeline runs end to end. When changes merge, the training pipeline runs on full data and registers a candidate model with its metrics. An evaluation gate compares it with the current production model on the same holdout set, and also checks slices, fairness and latency. If it passes, CD builds the serving image, deploys it to staging, runs integration and load tests, then rolls out with shadow or canary traffic while watching metrics. Bad signals trigger automatic rollback. For high-risk models I add a human approval step in the registry before production.

More medium MLOps Engineer questions

• Explain shadow, canary and A/B deployments for models. When do you use each?
• Design an automated retraining pipeline for a model whose data changes weekly.
• What operational challenges come with running a feature store, and how do you handle them?
• How do you run GPU model inference efficiently on Kubernetes?
• How do you design a training pipeline in an orchestrator like Airflow so that it is reliable and easy to rerun?
• How would you estimate the GPUs needed to serve an open-weight LLM for a given number of users?
• How would you define service level objectives for a production model service?
• How do you secure the ML pipeline and model supply chain?
• What would you monitor for an LLM-powered application in production, beyond normal service metrics?