Why use an orchestration tool like Airflow instead of cron jobs?

What the interviewer wants to hear

• Manages dependencies between tasks
• Retries, alerts and failure handling
• Backfills and parameterised runs by date
• Central visibility, logs and history

Model answer

Cron can start a script at a time, but it does not know that the transform job should only run after the extract finished successfully. An orchestrator like Airflow, Dagster or Prefect defines workflows as DAGs with explicit dependencies, so tasks run in the right order and downstream tasks wait or skip when something fails. It gives retries with backoff, timeouts and alerts. It also understands logical dates, which makes backfills easy: you can rerun a month of history with the right parameters. Everything is visible in one UI with logs, run history and durations, which helps debugging and on-call. With cron, all of this would be scattered scripts and custom code. Orchestrators also integrate with warehouses, Spark and cloud services through ready-made operators.

More easy Data Engineer questions

• Describe a data pipeline you built or would build, from source to dashboard.
• What is the difference between an OLTP database and an OLAP data warehouse?
• Explain a star schema and why it is used in data warehouses.
• What data quality checks would you add to a pipeline?
• What is a slowly changing dimension, and how do SCD Types 1, 2 and 3 differ?
• Why would you store analytics data as Parquet rather than CSV or JSON?
• What is change data capture (CDC), and why do data teams use it?
• Explain Kafka topics, partitions, offsets and consumer groups to a new teammate.
• What does dbt do in a modern data stack, and what are its main features?