hard20 pts · first answer countssparkspark-streamingexactly-once

Half-written files in a stream output

A streaming file sink writes Parquet to a folder. A Presto job reading the folder directly sometimes sees files from a failed batch. Spark readers do not. Why?

A.Presto reads files faster than Spark writes themB.The failed batch was written by the driverC.Spark reads the _spark_metadata log; Presto just lists filesD.Spark hides any file smaller than one MB

Discussion 0

Join free to post in the discussion.

Half-written files in a stream output

Discussion 0

Related questions