How do you secure the ML pipeline and model supply chain?

What the interviewer wants to hear

• Control and verify where models, data and packages come from
• Use safe model formats and scan or sign artifacts
• Pin images and dependencies by version or digest
• Least privilege for pipelines and serving, with managed secrets
• Audit trail from data to deployed model

Model answer

I treat models, datasets and packages as code that can be attacked. Third-party models come from approved sources and in safe formats like safetensors rather than pickle files, which can run code when loaded. Artifacts in the registry have hashes recorded and, ideally, signatures that the deployment checks. Docker images and Python dependencies are pinned and scanned, and deployments use image digests rather than mutable tags. Pipelines and serving jobs run with least-privilege service accounts, so a training job cannot write to production, and secrets come from a secrets manager. Training data access is logged, and sensitive data is masked where possible. Finally I keep an audit trail linking each deployed model to its code commit, data version, base model and approver, which helps both investigations and compliance.

More medium MLOps Engineer questions

• What does a good CI/CD pipeline for machine learning look like?
• Explain shadow, canary and A/B deployments for models. When do you use each?
• Design an automated retraining pipeline for a model whose data changes weekly.
• What operational challenges come with running a feature store, and how do you handle them?
• How do you run GPU model inference efficiently on Kubernetes?
• How do you design a training pipeline in an orchestrator like Airflow so that it is reliable and easy to rerun?
• How would you estimate the GPUs needed to serve an open-weight LLM for a given number of users?
• How would you define service level objectives for a production model service?
• What would you monitor for an LLM-powered application in production, beyond normal service metrics?