hard20 pts · first answer countssparkbackfillsidempotency

Overwrite deleted the whole table

A backfill job writes one day with df.write.mode("overwrite").partitionBy("dt").parquet(path) on a path-based table. Afterwards only that day exists and all other dates are gone. Why?

A.With the default static partitionOverwriteMode, overwrite replaces the whole output path; dynamic mode would replace only the partitions present in dfB.Parquet files expire after one dayC.partitionBy always deletes other partitionsD.The job ran out of memory and lost the data

Discussion 0

Join free to post in the discussion.

Overwrite deleted the whole table

Discussion 0

Related questions