easy5 pts · first answer countssparkspark-schema

Cost of inferSchema on CSV

A job reads a 200 GB CSV folder with inferSchema set to true. Why is this slower than giving a schema?

df = spark.read.option("header", True).option("inferSchema", True).csv("s3://raw/orders/")

A.inferSchema turns off parallel reading of the filesB.Spark makes an extra pass over the data to guess typesC.Spark converts the CSV to Parquet before readingD.Every column is read as a string twice

Discussion 0

Join free to post in the discussion.

Cost of inferSchema on CSV

Discussion 0

Related questions