Skip to content
RED SAGSkills

Apache Spark interview questions and answers

Spark interview questions focus on how distributed processing works under the hood and how to make jobs fast and stable.

Below are 60 Apache Spark questions with the correct answer and a short explanation, grouped from fresher to experienced level. Try each one before you open the answer.

Take the Apache Spark MCQ testPractise 136+ Apache Spark questions for points

What Apache Spark interviews usually check

  • ✓Transformations, actions and lazy evaluation
  • ✓Partitions, shuffles and data skew
  • ✓Joins, broadcast joins and caching
  • ✓Spark SQL and DataFrames
  • ✓Tuning memory, executors and streaming jobs

Apache Spark interview questions for freshers easy

1. What the Spark driver does

In a running Spark application, what is the main job of the driver?

  1. A.It stores every shuffle file for the cluster
  2. B.It builds the plan and schedules tasks on executors
  3. C.It runs all tasks of a stage by itself
  4. D.It keeps the only copy of cached data
Show answer

Answer: B. It builds the plan and schedules tasks on executors

The driver runs your main program, turns your code into a plan of stages and tasks, and hands tasks to executors. Executors do the actual data work and hold cached blocks and shuffle files. If the driver dies, the whole application stops.

2. Where a job splits into stages

Spark splits one job into several stages. What decides where one stage ends and the next begins?

  1. A.A shuffle, where data moves between partitions
  2. B.Each new line of code in the driver program
  3. C.Every filter or select step in the plan
  4. D.A fixed limit of 100 tasks in each stage
Show answer

Answer: A. A shuffle, where data moves between partitions

Narrow operations such as filter, select and withColumn are joined into one stage because each partition can be processed alone. A wide operation such as groupBy or a sort-merge join needs data from many partitions, so Spark ends the stage, writes shuffle files and starts a new stage that reads them.

3. Adding a constant column

You want to add a column country with the value "AE" on every row. Which PySpark code does this?

  1. A.df.withColumn("country", F.lit("AE"))
  2. B.df.withColumn("country", "AE")
  3. C.df.withColumn("country", F.col("AE"))
  4. D.df.select("country" == "AE")
Show answer

Answer: A. df.withColumn("country", F.lit("AE"))

withColumn needs a Column expression, and F.lit wraps a plain Python value as a constant column. Passing the string "AE" directly fails, and F.col("AE") looks for a column named AE. lit is also how you add fixed values like a load date or source name.

4. What explode() does

An order row has items = ["pen", "book", "bag"]. After select("order_id", explode("items")), how many rows does that order produce?

  1. A.1
  2. B.4
  3. C.3
  4. D.0
Show answer

Answer: C. 3

explode creates one output row for each element of the array and repeats the other columns. Three items give three rows. A row whose array is empty or null produces no rows with explode; explode_outer keeps it with a null.

5. union() with columns in another order

df_a has columns (id, name) and df_b has (name, id), both strings. What does df_a.union(df_b) do?

  1. A.Matches columns by name and lines them up correctly
  2. B.Fails because the column order is different
  3. C.Drops the columns that are in a different order
  4. D.Stacks rows by position, so names land in the id column
Show answer

Answer: D. Stacks rows by position, so names land in the id column

union, like SQL UNION ALL, matches columns by position, not by name. When the types are the same, it silently mixes up the data. Use unionByName to match by name, and allowMissingColumns=True if one side lacks some columns.

6. Customers with no orders

You want customers who have never placed an order. Which join type returns exactly those customer rows?

customers.join(orders, "customer_id", how=???)
  1. A."left_semi"
  2. B."left_anti"
  3. C."inner"
  4. D."cross"
Show answer

Answer: B. "left_anti"

A left anti join keeps rows from the left side that have no match on the right, and returns only left columns. A left semi join does the opposite: it keeps left rows that do have a match. Inner keeps matches and repeats rows for many orders.

7. Writing to a path that exists

You run df.write.parquet("s3://lake/sales/") without setting a mode, and the path already has data. What happens?

  1. A.The new data is appended to the old files
  2. B.The old data is replaced with the new data
  3. C.Spark writes to a new folder with a suffix
  4. D.The write fails with an error that the path exists
Show answer

Answer: D. The write fails with an error that the path exists

The default save mode is errorifexists, which stops you from overwriting data by accident. You must choose mode("append"), mode("overwrite") or mode("ignore") on purpose. Being explicit about the mode is a good habit in every pipeline.

8. What master local[*] means

You start Spark on your laptop with master set to local[*]. What does that mean?

  1. A.Spark connects to any cluster it finds on the network
  2. B.Spark runs one task at a time on one core
  3. C.Driver and executor run in one process using all cores
  4. D.Spark runs without a driver to save memory
Show answer

Answer: C. Driver and executor run in one process using all cores

Local mode runs Spark in a single JVM on one machine. The number in brackets is how many worker threads to use, and * means one per logical core. It is ideal for tests and small experiments, not for production data volumes.

9. Reading a pretty-printed JSON file

spark.read.json() on a file that holds one big pretty-printed JSON array gives a single _corrupt_record column. What fixes it?

  1. A.Set option("header", True)
  2. B.Set option("multiLine", True)
  3. C.Set option("inferSchema", False)
  4. D.Read it with spark.read.text()
Show answer

Answer: B. Set option("multiLine", True)

By default Spark expects JSON Lines: one complete JSON object per line, so each line can be parsed in parallel. A pretty-printed file spreads one object over many lines, so each line is invalid alone. multiLine=True parses whole files, but each file is then read by one task.

10. spark-submit in cluster mode

You run spark-submit --deploy-mode cluster from a laptop against a YARN cluster, then close the laptop. What happens to the job?

  1. A.It stops, because the driver process was on the laptop
  2. B.It pauses until the laptop reconnects
  3. C.It restarts from the beginning on another node
  4. D.It keeps running, as the driver runs inside the cluster
Show answer

Answer: D. It keeps running, as the driver runs inside the cluster

In cluster mode the driver is launched on a node in the cluster, so the submitting machine can go away. In client mode the driver runs where you submitted, so closing that machine kills the job. Scheduled production jobs normally use cluster mode.

11. Orders lost after explode

After exploding the items array, orders with an empty or null items array disappear from the result. Which function keeps them with a null item?

  1. A.posexplode
  2. B.flatten
  3. C.explode_outer
  4. D.array_compact
Show answer

Answer: C. explode_outer

explode drops rows whose array is null or empty because there is nothing to emit. explode_outer emits one row with a null instead, like a left outer join. posexplode also adds the element position but still drops empty arrays unless you use posexplode_outer.

12. Filling nulls per column

You want nulls in discount set to 0 and nulls in channel set to "unknown". Which call does this?

  1. A.df.fillna({"discount": 0, "channel": "unknown"})
  2. B.df.fillna([0, "unknown"], ["discount", "channel"])
  3. C.df.dropna(subset=["discount", "channel"])
  4. D.df.replace({None: 0, None: "unknown"})
Show answer

Answer: A. df.fillna({"discount": 0, "channel": "unknown"})

fillna (also df.na.fill) takes a dict that maps column names to replacement values, so each column can get its own default. A single value only fills columns of a matching type. dropna removes rows instead of filling them.

13. Scope of a temp view

You call df.createOrReplaceTempView("orders_today") in a job. Which code can query it with spark.sql?

  1. A.Any Spark job in the same workspace
  2. B.Only code using the same SparkSession
  3. C.Any user of the Hive metastore
  4. D.Any session until the cluster restarts
Show answer

Answer: B. Only code using the same SparkSession

A temporary view is just a named plan stored in the current SparkSession; it is not saved anywhere. When the session ends it is gone. A global temp view is shared across sessions in the same application under global_temp, and a real table is needed to share across jobs.

14. DROP TABLE on a managed table

You run DROP TABLE sales on a managed (internal) table in the metastore. What happens to its data files?

  1. A.They stay in place; only metadata is removed
  2. B.They move to a recycle bin for 30 days
  3. C.They are archived to cheaper storage
  4. D.They are deleted along with the table metadata
Show answer

Answer: D. They are deleted along with the table metadata

For a managed table, the catalog owns both the metadata and the data, so DROP TABLE deletes the files too. For an external table, created with a LOCATION, DROP removes only the metadata. Use external tables when other tools also own or read the files.

15. Fast count of unique visitors

A dashboard needs daily unique visitors on 5 billion events, and 1–2% error is fine. Which function is the cheapest choice?

  1. A.countDistinct
  2. B.collect_set then size
  3. C.approx_count_distinct
  4. D.distinct().count() per day
Show answer

Answer: C. approx_count_distinct

approx_count_distinct uses HyperLogLog++, which keeps a small fixed-size sketch per group instead of every distinct value. countDistinct and collect_set must keep or shuffle all the values. You can set the rsd argument to trade accuracy for memory.

16. Cut-off values in show()

df.show() prints addresses cut off with "...". Why, and how do you see the full text?

  1. A.The data was cut when written; reload it
  2. B.Strings over 20 characters are stored as nulls
  3. C.show() samples text; use collect() for the full value
  4. D.show() truncates to 20 characters; use truncate=False
Show answer

Answer: D. show() truncates to 20 characters; use truncate=False

By default show() prints 20 rows and truncates each value to 20 characters for a readable table. show(n, truncate=False) prints full values, and vertical=True helps with wide rows. The stored data is not changed.

17. filter() versus where()

What is the difference between df.filter(col("amount") > 1000) and df.where(col("amount") > 1000)?

  1. A.where runs on the driver, filter on executors
  2. B.None; where is an alias of filter
  3. C.filter keeps nulls, where removes them
  4. D.where can only take a SQL string
Show answer

Answer: B. None; where is an alias of filter

where() is simply an alias of filter() and produces the same plan. Both accept a Column expression or a SQL string such as "amount > 1000". Use whichever reads better for your team.

18. Grouping sales by month

order_ts is a timestamp. Which expression gives the first moment of its month, for monthly grouping?

  1. A.F.month("order_ts")
  2. B.F.last_day("order_ts")
  3. C.F.dayofmonth("order_ts")
  4. D.F.date_trunc("month", "order_ts")
Show answer

Answer: D. F.date_trunc("month", "order_ts")

date_trunc("month", ts) sets the day to 1 and the time to midnight, so all timestamps in a month map to the same value, keeping the year. month() returns only 1–12, which mixes up March 2025 and March 2026. last_day returns the last date of the month.

19. avg and count with a null value

A column has three rows: 10, 20 and null. What do F.avg("x") and F.count("x") return?

  1. A.avg 10, count 3
  2. B.avg 15, count 2
  3. C.avg 10, count 2
  4. D.avg 15, count 3
Show answer

Answer: B. avg 15, count 2

Aggregate functions skip nulls. avg is (10 + 20) / 2 = 15 and count("x") counts only the 2 non-null values. count("*") or count(1) would return 3 because it counts rows, not values.

20. Rows from a cross join

A table of 1,000 stores is cross joined with a table of 2,000 products to build a store-product grid. How many rows does the result have?

  1. A.3,000
  2. B.2,000
  3. C.10,00,000
  4. D.20,00,000
Show answer

Answer: D. 20,00,000

A cross join pairs every row on the left with every row on the right, so the count is 1,000 × 2,000 = 20,00,000 (2 million). That is why Spark asks for an explicit crossJoin() and why accidental cartesian products blow up jobs.

Apache Spark intermediate interview questions medium

21. What causes a shuffle

In Apache Spark, which operation typically causes a shuffle (moving data across executors)?

  1. A.filter() on a column
  2. B.select() of a few columns
  3. C.groupBy().agg() on a key
  4. D.withColumn() adding a computed column
Show answer

Answer: C. groupBy().agg() on a key

groupBy needs all rows with the same key on the same partition, so Spark redistributes data across the network: a wide transformation. filter, select and withColumn are narrow transformations that work within each partition.

22. One 20 GB gzip file

A job reads a single 20 GB file named events.csv.gz. The read stage has only one task and takes an hour. Why?

  1. A.Gzip cannot be split, so one task must read the whole file
  2. B.Spark reads compressed files on the driver only
  3. C.CSV files are always read by one task each
  4. D.The cluster has only one executor core free
Show answer

Answer: A. Gzip cannot be split, so one task must read the whole file

Gzip is not a splittable codec: to read byte 10 GB you must decompress everything before it. So Spark gives the whole file to one task. Ask for many smaller files, a splittable format like Parquet, or repartition right after reading so later stages are parallel.

23. partitionBy on user_id

A developer writes a table with partitionBy("user_id") for 5 million users. What problem follows?

  1. A.Spark refuses to partition by a string column
  2. B.All data goes to one folder because of hashing
  3. C.Millions of folders of tiny files that are slow to read
  4. D.Queries that filter by date become much faster
Show answer

Answer: C. Millions of folders of tiny files that are slow to read

partitionBy creates a folder for every distinct value, so a high-cardinality column creates millions of folders and small files. Listing and opening them dominates query time. Partition by low-cardinality columns that queries filter on, such as date, and use clustering or bucketing for ids.

24. Filtering on a partition column

A table is written with partitionBy("dt"), one folder per day for 3 years. A query filters WHERE dt = '2026-03-01'. What does Spark read?

  1. A.All files, then drops other dates in memory
  2. B.Only the first file of each folder
  3. C.All files, but only the dt column from each
  4. D.Only the files in the dt=2026-03-01 folder
Show answer

Answer: D. Only the files in the dt=2026-03-01 folder

This is partition pruning: the filter matches the folder structure, so Spark skips other folders without opening them. Here it reads 1 of about 1,095 days. Pruning works when the filter is on the partition column itself; wrapping it in a UDF can stop it.

25. A UDF in the WHERE clause

A filter on a Parquet table uses a Python UDF: df.filter(is_gulf(col("country"))). Compared with col("country").isin("AE", "SA"), what is lost?

  1. A.The query can no longer run in parallel tasks
  2. B.The filter cannot be pushed down to the Parquet scan
  3. C.Spark has to cache the table before the filter
  4. D.Column pruning stops working for all columns
Show answer

Answer: B. The filter cannot be pushed down to the Parquet scan

Catalyst cannot see inside a UDF, so it cannot push the filter into the Parquet reader to skip row groups with min/max stats. All rows are read and sent to Python first. Built-in expressions like isin can be pushed down and run in the JVM.

26. Reading SortMergeJoin in explain()

df.explain() shows SortMergeJoin with an Exchange hashpartitioning on both sides. What does that tell you?

  1. A.Both tables are shuffled by the join key and sorted before joining
  2. B.The smaller table is copied to every executor
  3. C.The join runs on the driver after collecting both sides
  4. D.Spark found the data already partitioned and skipped the shuffle
Show answer

Answer: A. Both tables are shuffled by the join key and sorted before joining

Exchange hashpartitioning means a shuffle by the join key, and SortMergeJoin sorts each side and merges matching keys. It is the default for two large tables. If one side is small, a broadcast hint or a higher threshold can replace both shuffles with a BroadcastHashJoin.

27. Bad-row counter that runs high

An accumulator counts bad rows inside a map() transformation. Some tasks failed and were retried, and one stage ran twice. The count is higher than the real number of bad rows. Why?

  1. A.Accumulators are not safe across more than one executor
  2. B.Updates inside transformations can repeat when tasks rerun
  3. C.The driver adds each value twice by design
  4. D.Accumulators reset to zero only at job start
Show answer

Answer: B. Updates inside transformations can repeat when tasks rerun

Spark only guarantees that accumulator updates made inside actions such as foreach are applied once per task. Inside transformations, retried tasks, speculative copies and recomputed stages all add again. Use accumulators in transformations only for rough debugging, or count bad rows with a normal aggregation.

28. A 50 MB Python dict in every task

A UDF uses a 50 MB Python dict of GST rates that is referenced in the function closure. Each of 2,000 tasks seems to ship it again. What should you use?

  1. A.An accumulator that holds the dict
  2. B.A global variable defined on the driver
  3. C.A cache() call on the dict before the UDF
  4. D.A broadcast variable, sent once to each executor
Show answer

Answer: D. A broadcast variable, sent once to each executor

Objects captured in a closure are serialized with every task. sc.broadcast(rates) sends the value once per executor, where it is cached, and tasks read it with rates.value. Better still, if the lookup is a table, turn it into a DataFrame and use a broadcast join.

29. Rows with null keys vanish in a join

Two tables are inner-joined on col("a.ref") == col("b.ref"). Both have rows where ref is null, but none of them appear in the result. Which condition would match null to null?

  1. A.col("a.ref").isNull() == col("b.ref")
  2. B.col("a.ref").eqNullSafe(col("b.ref"))
  3. C.col("a.ref") === col("b.ref") with F.lit(None)
  4. D.col("a.ref").contains(col("b.ref"))
Show answer

Answer: B. col("a.ref").eqNullSafe(col("b.ref"))

In SQL logic, null = null is unknown, not true, so null keys never match with ==. eqNullSafe, or <=> in Spark SQL, treats two nulls as equal. Often the better fix is to decide what a null key means and clean it before the join.

30. Structured Streaming output modes

A streaming query counts orders per city. Which output mode writes the whole updated result table to the sink on every trigger?

  1. A.complete
  2. B.append
  3. C.update
  4. D.overwrite
Show answer

Answer: A. complete

complete mode outputs every row of the result table each trigger, which works only for aggregations and small results. update outputs only rows that changed in this trigger. append outputs rows once, when they are final, which for aggregations needs a watermark.

31. Sorting rows inside each file

You only need rows sorted by event_time inside each output file to help compression, not a global order. Which is cheaper?

  1. A.sortWithinPartitions("event_time")
  2. B.orderBy("event_time")
  3. C.sort("event_time").coalesce(1)
  4. D.repartitionByRange(1, "event_time")
Show answer

Answer: A. sortWithinPartitions("event_time")

orderBy does a global sort: Spark samples the data and range-partitions it with a shuffle. sortWithinPartitions sorts each existing partition locally with no shuffle. For better min/max stats and compression in each file, a local sort is enough.

32. Removing idle executors safely

Dynamic allocation removes idle executors. What must be in place so that removing one does not lose shuffle files later stages need?

  1. A.An external shuffle service or shuffle tracking
  2. B.A cache() call on every shuffled DataFrame
  3. C.A fixed spark.executor.instances value
  4. D.Speculative execution turned on
Show answer

Answer: A. An external shuffle service or shuffle tracking

Shuffle files live on the executor that wrote them. An external shuffle service serves them after the executor is gone, while shuffle tracking (common on Kubernetes) keeps executors with needed shuffle data alive. Without either, removed executors would cause fetch failures and recomputation.

33. One file per date folder

You want about one file in each dt= folder when writing with partitionBy("dt"). Which line goes before the write?

  1. A.df = df.repartition("dt")
  2. B.df = df.coalesce(1)
  3. C.df = df.sortWithinPartitions("dt")
  4. D.df = df.repartition(200)
Show answer

Answer: A. df = df.repartition("dt")

repartition("dt") hash-partitions by dt, so all rows of one date sit in one task, which writes one file into that folder. coalesce(1) writes everything from one task, losing parallelism. repartition(200) spreads each date across many tasks, creating many files per folder.

34. Reading yesterday's table version

A bad job ran at 10:00 on a Delta table. You want to read the table exactly as it was at version 41, before that job. How?

  1. A.spark.read.option("versionAsOf", 41).table("sales")
  2. B.spark.read.table("sales").limit(41)
  3. C.spark.read.parquet("sales/_delta_log/41")
  4. D.spark.sql("SELECT * FROM sales WHERE version = 41")
Show answer

Answer: A. spark.read.option("versionAsOf", 41).table("sales")

Delta time travel reads an older snapshot with versionAsOf or timestampAsOf, or SQL VERSION AS OF 41. DESCRIBE HISTORY shows the versions and which operation made each. RESTORE TABLE can roll the table back, as long as VACUUM has not removed the old files.

35. Customers who ordered, no duplicates

You need customers who have at least one order, one row per customer, with only customer columns. An inner join gives duplicates. Which join fits best?

  1. A.A left outer join
  2. B.A left semi join
  3. C.A full outer join
  4. D.A cross join with a filter
Show answer

Answer: B. A left semi join

A left semi join returns each left row at most once if any match exists, and returns only left columns, like WHERE EXISTS. An inner join repeats a customer once per order. Semi joins can also be cheaper because Spark only needs to check for existence.

36. Same timestamp, different hour

A team in Dubai and a team in Bengaluru run the same query on a timestamp column and see different hours. Nothing is wrong with the data. What explains it?

  1. A.Parquet stores a separate copy of times per region
  2. B.The cluster clock is wrong on one of the clusters
  3. C.Each session shows times in its own session time zone
  4. D.Timestamps lose their hours when read from Parquet
Show answer

Answer: C. Each session shows times in its own session time zone

Spark TIMESTAMP values are stored as instants in UTC and shown in the session time zone. Asia/Dubai is UTC+4 and Asia/Kolkata is UTC+5:30, so the same instant shows hours 1.5 apart. Set the session time zone on purpose, and use TIMESTAMP_NTZ for wall-clock values without a zone.

37. Parsing a JSON string column

A Kafka value column holds JSON text. Which is the usual way to turn it into typed columns in Spark?

  1. A.col("value").cast("struct<data:string>")
  2. B.from_json(col("value").cast("string"), schema)
  3. C.explode(col("value").cast("array"))
  4. D.F.to_json(col("value").cast("string"))
Show answer

Answer: B. from_json(col("value").cast("string"), schema)

from_json parses a JSON string column with a schema you provide and returns a struct, which you can expand with select("data.*"). Kafka values are binary, so cast to string first. Invalid JSON gives null, so count nulls as a data quality check.

38. Thousands of tiny Delta files

A Delta table written by a streaming job has 80,000 files of about 200 KB each. Queries usually filter on customer_id. What is the standard fix?

  1. A.Run OPTIMIZE with ZORDER BY (customer_id)
  2. B.Run VACUUM with 0 hours retention
  3. C.Repartition the table by customer_id folders
  4. D.Convert the table back to plain Parquet
Show answer

Answer: A. Run OPTIMIZE with ZORDER BY (customer_id)

OPTIMIZE rewrites many small files into fewer large ones, and ZORDER BY clusters rows with similar customer_id together so file-level min/max stats skip more files. VACUUM only deletes unreferenced old files. Partitioning by a high-cardinality id makes the small-file problem worse.

39. Where to fit a StandardScaler

An ML engineer builds a PySpark MLlib model. Which order avoids leaking test data into training?

  1. A.Fit the scaler on all data, then split, then train
  2. B.Split, fit the scaler on test, then transform train
  3. C.Fit the scaler and model on all data, then split
  4. D.Split, then fit a Pipeline (scaler + model) on train
Show answer

Answer: D. Split, then fit a Pipeline (scaler + model) on train

The scaler learns mean and standard deviation, so fitting it on all rows lets test statistics influence training. Putting the scaler and model in a Pipeline and calling fit on the training split keeps every learned value from train only; the fitted PipelineModel then transforms test.

40. Window sum when days repeat

With only orderBy in the window, what running_total values come out for these four rows (ordered as shown)?

# day, amount
# 1, 100
# 2, 50
# 2, 30
# 3, 20
w = Window.orderBy("day")
df.withColumn("running_total", F.sum("amount").over(w))
  1. A.100, 150, 180, 200
  2. B.100, 50, 30, 20
  3. C.100, 180, 180, 200
  4. D.200, 200, 200, 200
Show answer

Answer: C. 100, 180, 180, 200

When a window has orderBy but no explicit frame, Spark uses RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. RANGE includes all peer rows with the same day, so both day-2 rows get 100 + 50 + 30 = 180. Use rowsBetween(Window.unboundedPreceding, 0) with a tie-breaker for a row-by-row total.

Apache Spark interview questions for experienced candidates hard

41. Sample changes on every run

A training sample is built with this Spark code. Two runs on the same data give different rows. Why?

sample = (spark.read.parquet("s3://data/events/")
          .filter("country = 'IN'")
          .limit(100000))
  1. A.Spark always randomises rows unless cached
  2. B.filter() changes results based on cluster size
  3. C.limit without orderBy returns whichever rows arrive first
  4. D.Spark cannot read Parquet deterministically
Show answer

Answer: C. limit without orderBy returns whichever rows arrive first

Without an explicit ordering, the rows returned by limit depend on partition and task timing. For reproducible samples, sort by a stable key first, or use sample() with a fixed seed on a fixed input version.

42. Which late event is kept

A Spark Structured Streaming job counts events in 5-minute event-time windows with withWatermark("event_time", "10 minutes"). The maximum event time seen so far is 12:31. Two late events now arrive: one at 12:18 and one at 12:23. Which is guaranteed to be counted?

  1. A.Both events
  2. B.Neither event
  3. C.Only the 12:23 event
  4. D.Only the 12:18 event
Show answer

Answer: C. Only the 12:23 event

The watermark is max event time minus the delay: 12:31 - 10 minutes = 12:21. The 12:23 event is after the watermark, so its window [12:20, 12:25) is still open and it is counted. The 12:18 event belongs to window [12:15, 12:20), which ends before the watermark, so Spark may drop it; data older than the watermark is not guaranteed to be processed.

43. A slow Python UDF in PySpark

A PySpark job applies a row-by-row Python UDF to clean phone numbers and is 10 times slower than the rest of the pipeline. What is the best first fix?

  1. A.Rewrite it with built-in Spark SQL functions (such as regexp_replace), or a vectorised pandas UDF if logic is complex
  2. B.Call collect() and clean the data on the driver
  3. C.Add cache() after the UDF
  4. D.Increase the number of retries
Show answer

Answer: A. Rewrite it with built-in Spark SQL functions (such as regexp_replace), or a vectorised pandas UDF if logic is complex

Plain Python UDFs move every row between the JVM and Python workers and are opaque to the Catalyst optimiser. Built-in functions run inside the JVM and are optimised. When Python is needed, Arrow-based pandas UDFs process whole batches and cut serialisation overhead.

44. Overwrite deleted the whole table

A backfill job writes one day with df.write.mode("overwrite").partitionBy("dt").parquet(path) on a path-based table. Afterwards only that day exists and all other dates are gone. Why?

  1. A.With the default static partitionOverwriteMode, overwrite replaces the whole output path; dynamic mode would replace only the partitions present in df
  2. B.Parquet files expire after one day
  3. C.partitionBy always deletes other partitions
  4. D.The job ran out of memory and lost the data
Show answer

Answer: A. With the default static partitionOverwriteMode, overwrite replaces the whole output path; dynamic mode would replace only the partitions present in df

In static mode, which is the Spark default, an overwrite clears all existing partitions under the target. Setting spark.sql.sources.partitionOverwriteMode=dynamic replaces only the partitions being written. Table formats offer safer options such as Delta replaceWhere or Iceberg overwrite by filter.

45. Broadcast hint on the wrong side

A large orders table is left-outer-joined to a small cities table, but the hint is put on orders: broadcast(orders).join(cities, "city_id", "left"). Spark ignores it. Why?

  1. A.Broadcast hints work only in Spark SQL, not in Python
  2. B.Spark never broadcasts a table that has a city_id key
  3. C.Hints are ignored whenever AQE is turned on
  4. D.A left outer join can only broadcast the right side
Show answer

Answer: D. A left outer join can only broadcast the right side

For a left outer join, every row from the left side must appear even without a match, so Spark streams the left side and can only build a hash table from the right side. A hint on the left side cannot be honoured. Put the broadcast on cities instead, which is also the small table here.

46. JDBC read with lowerBound and upperBound

A JDBC read uses partitionColumn="id", lowerBound=1, upperBound=1000000, numPartitions=10. The table has ids up to 5 million. What happens?

  1. A.Only rows with ids from 1 to 1,000,000 are read
  2. B.All rows are read; ids above 1,000,000 go into the last partition
  3. C.The read fails because ids exceed upperBound
  4. D.Spark splits 5 million ids evenly into 10 partitions
Show answer

Answer: B. All rows are read; ids above 1,000,000 go into the last partition

lowerBound and upperBound only decide the stride of the ranges; they do not filter. The first and last partitions are open-ended, so every id above 1,000,000 lands in the last partition and makes it huge. Set the bounds from the real MIN and MAX of the column.

47. Unified memory for a 10 GB heap

An executor has a 10 GB heap (10,240 MB). Spark reserves 300 MB, and spark.memory.fraction is 0.6. About how much unified memory (execution plus storage) does it get?

  1. A.About 6,144 MB
  2. B.About 3,072 MB
  3. C.About 5,964 MB
  4. D.About 9,940 MB
Show answer

Answer: C. About 5,964 MB

Unified memory = (heap - 300 MB reserved) x 0.6 = (10,240 - 300) x 0.6 = 9,940 x 0.6 = 5,964 MB. Execution and storage share this pool and can borrow from each other. The remaining 40% is user memory for your own objects and Spark internals.

48. Container killed for exceeding memory

A PySpark job on YARN fails with "Container killed by YARN for exceeding memory limits". The JVM heap usage in the UI looks fine. What should you raise first?

  1. A.spark.sql.shuffle.partitions
  2. B.spark.driver.maxResultSize
  3. C.spark.memory.storageFraction
  4. D.spark.executor.memoryOverhead
Show answer

Answer: D. spark.executor.memoryOverhead

YARN kills the whole container when total process memory goes over its limit, including off-heap memory, network buffers and Python worker processes, which all live outside the JVM heap. That non-heap space is set by memoryOverhead (default 10% or 384 MB). Raising heap does not help if the overhead is what overflows.

49. Adding 300 columns with withColumn

Code adds 300 derived columns with a for loop that calls df = df.withColumn(...) each time. The job spends minutes before any task starts. What is the better pattern?

  1. A.Cache the DataFrame after every withColumn call
  2. B.Use more shuffle partitions for the job
  3. C.Build all expressions and add them in one select()
  4. D.Run the loop on the executors with mapPartitions
Show answer

Answer: C. Build all expressions and add them in one select()

Each withColumn creates a new projection in the plan, so 300 calls create a deep plan that the analyzer and optimizer must walk again and again. Building a list of column expressions and calling select once, or withColumns with a dict in Spark 3.3+, creates one projection.

50. startingOffsets changed but ignored

A Structured Streaming query reads Kafka with a checkpoint. You change startingOffsets from "latest" to "earliest" and restart to reprocess history. Nothing old is read. Why?

  1. A.Kafka deletes old data whenever a consumer restarts
  2. B.On restart, offsets come from the checkpoint, not startingOffsets
  3. C.earliest is only allowed for batch reads
  4. D.Spark needs the topic to be recreated first
Show answer

Answer: B. On restart, offsets come from the checkpoint, not startingOffsets

startingOffsets applies only when a query starts with no checkpoint. After that, the offsets saved in the checkpoint decide where to resume, which is what gives exactly-once recovery. To reprocess, start with a new checkpoint location, and make sure the sink can handle the replay.

51. Stream-stream join running out of memory

A streaming join of ad impressions to clicks runs for days and its state store keeps growing until executors fail. What is missing?

  1. A.A larger trigger interval such as 10 minutes
  2. B.A cache() call on both input streams
  3. C.Watermarks plus a time bound in the join condition
  4. D.Complete output mode for the join query
Show answer

Answer: C. Watermarks plus a time bound in the join condition

Without a watermark and an event-time range in the join condition, Spark must keep every past row in case a future match arrives, so state grows forever. With them, for example clicks within 1 hour of an impression, Spark can drop old state. This bounds memory at the cost of ignoring very late events.

52. Pruning a fact table through a join

A fact table partitioned by date_key is joined to a date dimension filtered to one quarter. The query has no direct filter on the fact table, yet Spark reads only about 90 date partitions. What did this?

  1. A.Static partition pruning from the WHERE clause
  2. B.Dynamic partition pruning using the dimension filter
  3. C.Column pruning in the Parquet reader
  4. D.AQE coalescing the shuffle partitions
Show answer

Answer: B. Dynamic partition pruning using the dimension filter

With dynamic partition pruning, Spark first runs the filtered dimension side and uses its join keys as a runtime filter on the fact table scan. Only matching partitions are read. It works best when the dimension side is small enough to broadcast.

53. A broadcast hint on 8 GB

Someone adds broadcast() to an 8 GB table to speed up a join. The job now fails with driver out-of-memory or broadcast timeout errors. Why?

  1. A.Broadcast data is written to S3 and read back slowly
  2. B.Spark refuses to broadcast tables with string columns
  3. C.The table is gathered on the driver, then sent to every executor
  4. D.The hint forces the other table to be broadcast as well
Show answer

Answer: C. The table is gathered on the driver, then sent to every executor

A broadcast join builds the small side on the driver and ships a full copy to each executor, which must hold it in memory. That only works for tables up to a few hundred MB, and Spark caps broadcasts at 8 GB. For two large tables, use sort-merge join, bucketing or pre-aggregation.

54. Half-written files in a stream output

A streaming file sink writes Parquet to a folder. A Presto job reading the folder directly sometimes sees files from a failed batch. Spark readers do not. Why?

  1. A.Presto reads files faster than Spark writes them
  2. B.The failed batch was written by the driver
  3. C.Spark reads the _spark_metadata log; Presto just lists files
  4. D.Spark hides any file smaller than one MB
Show answer

Answer: C. Spark reads the _spark_metadata log; Presto just lists files

The file sink records each committed batch in a _spark_metadata log, and Spark readers use it to see only committed files, which gives exactly-once output. Other engines just list the folder and can see leftover files from failed attempts. Writing to Delta or Iceberg gives every engine a proper commit log.

55. Using spot instances for Spark

To cut cloud cost by about 60%, a team plans to move a nightly Spark cluster to spot instances. Which set-up is the usual safe choice?

  1. A.Driver on spot, executors on on-demand
  2. B.Driver on on-demand, executors mostly on spot
  3. C.Everything on spot with speculation turned off
  4. D.Everything on spot with one large executor
Show answer

Answer: B. Driver on on-demand, executors mostly on spot

Losing the driver kills the whole application, so it should run on stable on-demand capacity. Losing a spot executor only costs recomputing its tasks and shuffle output, which Spark handles automatically. Graceful decommissioning can also migrate shuffle and cache blocks when spot nodes get a warning.

56. Which partition AQE treats as skewed

AQE skew join is on with default settings (skewedPartitionFactor 5, skewedPartitionThresholdInBytes 256 MB). The median shuffle partition is 40 MB. Which partition will AQE split?

  1. A.A 180 MB partition
  2. B.A 240 MB partition
  3. C.A 300 MB partition
  4. D.None of these sizes
Show answer

Answer: C. A 300 MB partition

A partition counts as skewed only if it is larger than 5 × median (200 MB) AND larger than 256 MB. 180 MB fails both tests and 240 MB passes the first but not the second. 300 MB passes both, so AQE splits it into smaller tasks.

57. Cost of salting a skewed join

One merchant_id holds 40% of a 2 billion row fact table. You salt the fact side with a random salt 0–7 and join to a 2 million row merchant table. How many rows must the merchant side have after salting?

  1. A.2 million, unchanged
  2. B.16 million
  3. C.250,000
  4. D.8 million
Show answer

Answer: B. 16 million

Each fact row gets one of 8 salts, so every merchant row must exist once per salt to still match. The merchant side is exploded 8 times: 2 million × 8 = 16 million rows. This is the price of salting, so salt only the hot keys when the small side is large.

58. Rows produced by cube on two columns

Sales data has 3 cities and 4 channels, and every city-channel pair has sales. How many rows does df.cube("city", "channel").sum("amount") return?

  1. A.12
  2. B.16
  3. C.19
  4. D.20
Show answer

Answer: D. 20

cube produces every grouping set: (city, channel) gives 12 rows, (city) gives 3, (channel) gives 4, and the grand total gives 1. That is 12 + 3 + 4 + 1 = 20. rollup would skip the channel-only level and return 16.

59. Time travel fails after VACUUM

A Delta table is updated daily. After VACUUM runs with the default retention, a query using VERSION AS OF a 30-day-old version fails. Why?

  1. A.VACUUM deleted data files only that old version used
  2. B.VACUUM deletes the transaction log for all versions
  3. C.Time travel only works for the last 3 versions
  4. D.VERSION AS OF needs the table to be cached
Show answer

Answer: A. VACUUM deleted data files only that old version used

VACUUM removes data files no longer referenced by the current table and older than the retention period (7 days by default). The log still lists the old version, but its files are gone, so reading it fails. Keep retention long enough for any time travel or audit you need.

60. Streaming state causing long GC pauses

A Structured Streaming dedup query keeps state for 50 million keys. Executors show long GC pauses and batches slow down. Which change targets the cause?

  1. A.Switch the output mode from append to complete
  2. B.Use the RocksDB state store provider
  3. C.Lower spark.sql.shuffle.partitions to 10
  4. D.Remove the watermark so state is never dropped
Show answer

Answer: B. Use the RocksDB state store provider

The default state store keeps state as JVM objects on the heap, so huge state means heavy garbage collection. The RocksDB provider keeps state in native memory and local disk, outside the JVM heap. A watermark is still needed so old keys are eventually dropped.

Ready to test yourself?

Take a 15-question Apache Spark MCQ test with a score at the end, or practise the full Apache Spark question bank and earn points on the leaderboard.

Start the Apache Spark testTake a mock interview