hard20 pts · first answer countsairfloworchestrationairflow-best-practices

Workers dying on a pandas join

A @task loads two 30 GB tables into pandas on an Airflow worker to join them, and the worker keeps getting OOM-killed. What is the best design change?

A.Give the worker 4 more GB of memoryB.Run the join in the warehouse or Spark and orchestrate it from AirflowC.Split the task into two tasks that share a global DataFrameD.Set retries=10 so it eventually passes

Discussion 0

Join free to post in the discussion.

Workers dying on a pandas join

Discussion 0

Related questions