hard20 pts · first answer countssparkparquetschema-evolution

A new column missing on read

Newer Parquet files in a folder have an extra column promo_code, older ones do not. spark.read.parquet(folder) sometimes shows no promo_code column. Why?

A.Spark takes the schema from some files unless mergeSchema is onB.Parquet cannot store columns added after the first fileC.Spark drops any column that has null valuesD.The promo_code files are skipped as corrupt

Discussion 0

Join free to post in the discussion.

A new column missing on read

Discussion 0

Related questions