medium10 pts · first answer countssparkperformance

When to cache in Spark

In which case does calling df.cache() help most?

A.A DataFrame read once and written onceB.A DataFrame that is expensive to compute and is reused by several actions in the same jobC.A DataFrame larger than total cluster memory that is used onceD.Before every single transformation, to be safe

← When a daily Airflow run starts repartition vs coalesce →