medium10 pts · first answer countsairflowairflow-concurrency

Limiting one heavy task across runs

Several runs of a DAG can be active at once, but its heavy export task must never run more than 2 copies at the same time. Which setting is meant for this?

A.max_active_runs=2 on the DAGB.max_active_tis_per_dag=2 on the taskC.retries=2 on the taskD.parallelism=2 in airflow.cfg

Discussion 0

Join free to post in the discussion.

Limiting one heavy task across runs

Discussion 0

Related questions