hard20 pts · first answer countsml-pipelinessparkreproducibility

Sample changes on every run

A training sample is built with this Spark code. Two runs on the same data give different rows. Why?

sample = (spark.read.parquet("s3://data/events/")
          .filter("country = 'IN'")
          .limit(100000))

A.Spark always randomises rows unless cachedB.filter() changes results based on cluster sizeC.limit without orderBy returns whichever rows arrive firstD.Spark cannot read Parquet deterministically

Discussion 0

Join free to post in the discussion.

Sample changes on every run

Discussion 0

Related questions