How do you design a training pipeline in an orchestrator like Airflow so that it is reliable and easy to rerun?

What the interviewer wants to hear

• Small tasks with clear inputs and outputs
• Idempotent tasks that write to partitioned, versioned locations
• Retries with backoff and timeouts
• Data validation and quality gates between steps
• Safe backfills with controlled concurrency

Model answer

I split the pipeline into small tasks such as extract, validate, build features, train, evaluate and register, each with clear inputs and outputs passed by reference to versioned storage. Every task is idempotent: it writes to a partition keyed by the run date and overwrites it, so rerunning never duplicates data. Tasks have retries with backoff for transient errors and timeouts so a hung job does not block the DAG. Between steps I add gates: data validation before training and a champion comparison before registration, so bad data or a worse model stops the pipeline. For backfills I keep catchup off by default, run them deliberately with a limited number of active runs, and make sure the source systems can handle the load. Finally, each run logs lineage, such as data versions, code commit and resulting model version, so any output can be traced.

More medium MLOps Engineer questions

• What does a good CI/CD pipeline for machine learning look like?
• Explain shadow, canary and A/B deployments for models. When do you use each?
• Design an automated retraining pipeline for a model whose data changes weekly.
• What operational challenges come with running a feature store, and how do you handle them?
• How do you run GPU model inference efficiently on Kubernetes?
• How would you estimate the GPUs needed to serve an open-weight LLM for a given number of users?
• How would you define service level objectives for a production model service?
• How do you secure the ML pipeline and model supply chain?
• What would you monitor for an LLM-powered application in production, beyond normal service metrics?