Pandas questions test everyday data work in Python: selecting, cleaning, joining and aggregating DataFrames, and avoiding slow loops.
Below are 60 Pandas questions with the correct answer and a short explanation, grouped from fresher to experienced level. Try each one before you open the answer.
What does this code print?
import pandas as pd
import numpy as np
df = pd.DataFrame({'a': [1, np.nan, 3], 'b': [np.nan, np.nan, 6]})
print(df.isna().sum().tolist())Answer: A. [1, 2]
isna() returns a True/False frame, and sum() adds up the True values column by column. Column a has one missing value and column b has two, so the result is [1, 2]. This is usually the first check when profiling a new dataset.
After pd.read_csv("orders.csv"), the order_date column holds strings, so .dt fails. What is the most direct fix at load time?
Answer: B. pd.read_csv("orders.csv", parse_dates=["order_date"])
parse_dates tells read_csv to convert the listed columns to datetime64 while reading. You can also convert after loading with pd.to_datetime(df["order_date"], format="%Y-%m-%d"), which is useful when you need an explicit format. "date" is not a valid dtype name for dtype.
Signups are sorted by time, oldest first, and some emails appear more than once. Which call keeps only the most recent row for each email?
Answer: A. df.drop_duplicates(subset=["email"], keep="last")
subset limits the comparison to the email column, and keep="last" keeps the final occurrence, which here is the newest. Without subset, rows are only duplicates when every column matches. keep=False would drop all copies, and duplicated only returns a boolean mask.
Which call sorts by city A to Z and, within each city, by sales from highest to lowest?
Answer: D. df.sort_values(["city", "sales"], ascending=[True, False])
sort_values accepts a list of columns and a matching list of directions, sorting by the first column and breaking ties with the next. Chaining two sorts orders the result mainly by sales, because the last sort wins and city only breaks ties at best. sort_index sorts by the index, not columns.
df["order_date"] is a datetime64 column. Which expression gives weekday names like "Friday"?
Answer: C. df["order_date"].dt.day_name()
Date parts of a datetime Series are reached through the .dt accessor, and day_name() returns the full weekday name. dt.weekday gives a number from 0 (Monday) to 6 (Sunday) instead. Calling day_name directly on the Series fails, because the method lives on the accessor.
You share a notebook and want everyone to get the same 10% sample of rows. Which call does that?
Answer: D. df.sample(frac=0.1, random_state=42)
frac picks a share of rows, and random_state fixes the random generator so the same rows come back each run. n expects a whole number of rows and seed is not a parameter. head would take the first rows, which is not random and often biased by sort order.
Which line keeps only rows where city is Dubai, Abu Dhabi or Sharjah?
Answer: B. df[df["city"].isin(["Dubai", "Abu Dhabi", "Sharjah"])]
isin returns True for each row whose value is in the list, giving a mask you can filter with. Comparing a Series to a list with == compares item by item and fails when lengths differ, and the Python in operator checks the index, not values. filter selects columns by label, not rows.
You want to remove only the rows where email is missing, and keep rows that miss other values. Which call?
Answer: B. df.dropna(subset=["email"])
subset restricts the check to the listed columns. Plain dropna drops a row if any column is missing, which can throw away far more data than intended. axis=1 drops columns, and how="all" drops only rows where every value is missing.
"Anil@Mail.com " and "anil@mail.com" are the same customer, but drop_duplicates keeps both. What should you do first?
Answer: A. df["email"] = df["email"].str.strip().str.lower()
Exact matching treats different spacing and capital letters as different values. Stripping spaces and lower-casing gives one standard form, and then drop_duplicates works. Normalising keys like this is a standard first cleaning step before joins and counts.
You save with df.to_csv("out.csv") and read it back; now there is an extra column called "Unnamed: 0". How do you avoid it?
Answer: B. Save with df.to_csv("out.csv", index=False)
to_csv writes the index as the first column by default, and read_csv turns that nameless column into "Unnamed: 0". index=False skips writing it. If the index carries meaning, save it and read it back with index_col=0 instead.
A training job that worked last month now fails or gives different results, though the code has not changed. The requirements file lists "scikit-learn" and "pandas" with no versions. What is the best fix?
Answer: A. Pin exact dependency versions (for example with a lock file) and build from a fixed base image
Unpinned dependencies install whatever version is newest at build time, so behaviour changes silently. Lock files and fixed base images make builds reproducible. Upgrades should be deliberate and tested.
requirements.txt lists "pandas>=1.5". A fresh install months later pulls a new major version and training code breaks. What prevents this?
Answer: D. A lockfile with exact versions, e.g. from uv or pip-compile
Ranges let installs change over time. A lockfile pins every direct and transitive package to an exact version (often with hashes), so the same environment is rebuilt later and on every machine.
Which pandas code shows how many missing values each column of the DataFrame df has?
Answer: B. df.isna().sum()
df.isna() gives True where a value is missing, and .sum() adds the Trues per column, so you get a count of missing values for each column. df.count() gives the number of non-missing values instead, and dropna() removes rows.
df has columns city and amount. Which line gives total amount per city?
Answer: C. df.groupby('city')['amount'].sum()
groupby('city') splits the rows into one group per city, ['amount'] picks the column, and .sum() adds it up for each group. It is the pandas version of SELECT city, SUM(amount) ... GROUP BY city.
What is a partition in a Spark DataFrame?
Answer: A. A chunk of the data that one task processes, which is the unit of parallelism
Spark splits a DataFrame into partitions and runs one task per partition on an executor core. Too few partitions leaves cores idle, while too many tiny partitions adds scheduling overhead. This is different from a table partition folder on disk, although the two are related.
A 40 GB CSV file must be processed in Python on a laptop with 16 GB of RAM. What is a sensible approach?
Answer: C. Read it in chunks (for example pd.read_csv with chunksize) or use an engine like DuckDB or Polars that streams the data
Loading the whole file at once will run out of memory. Processing it in chunks, or using an out-of-core engine like DuckDB or Polars lazy scans, keeps memory bounded. Converting it once to Parquet also makes later reads much faster.
A developer runs df.collect() on a DataFrame with about 500 GB of data to "look at it". What is the most likely result?
rows = df.collect() print(rows[:10])
Answer: A. The driver runs out of memory and fails
collect() pulls every row from every executor into the driver process. The driver usually has a few GB of memory, so a large result causes an out-of-memory error. Use show(), take(n) or limit(n) to look at a few rows instead.
A stage reads a DataFrame that has 64 partitions and does only narrow work such as filter and select. How many tasks does that stage run?
Answer: C. 64
Spark runs one task per partition in a stage. With 64 partitions, the stage has 64 tasks, which run in parallel as far as the available cores allow. Changing the partition count is how you change the parallelism of a stage.
An analyst knows pandas well and wants to run similar code on a 200 GB dataset using Spark. Which option fits best?
Answer: B. Use pyspark.pandas, the pandas API on Spark
pyspark.pandas (import pyspark.pandas as ps) gives a pandas-like API whose operations run as distributed Spark jobs. toPandas() pulls all 200 GB to the driver and will fail. Plain pandas in one task gets no parallelism.
A DataFrame has exactly 1,00,000 rows. You run df.sample(fraction=0.1).count(). What should you expect?
Answer: A. About 10,000 rows, not exactly
sample() without replacement keeps each row independently with probability 0.1, so the result is close to 10,000 but varies. Pass a seed for repeatable results. If you need an exact number of rows, use orderBy(rand()).limit(n) or similar.
You forecast tomorrow's demand. Which lag feature is safe to use?
df = df.sort_values('date')
# option A
df['lag_1'] = df['demand'].shift(1)
# option B
df['lead_1'] = df['demand'].shift(-1)Answer: D. A: shift(1), yesterday's value on each row
shift(1) puts the previous day's demand on each row, which is known at prediction time. shift(-1) pulls the next day's value backwards, which is the future and leaks the answer. In grouped data, shift within each store or product group.
What does this pandas code produce?
df["city_avg"] = df.groupby("city")["sales"].transform("mean")Answer: C. A new column where every row gets the average sales of its own city
transform returns a result with the same index and length as the original DataFrame, so each row receives its group statistic. agg or mean would return one row per city instead. This pattern is handy for features like "sales compared with city average".
s holds "M", "F" and "X". What do these two lines return?
codes = {"M": "Male", "F": "Female"}
a = s.map(codes)
b = s.replace(codes)Answer: D. a has NaN for "X", b keeps "X"
map looks every value up in the dict and gives NaN when a key is missing. replace only changes values found in the dict and leaves the rest alone. Use map when every value must be mapped, and check for new NaNs afterwards to catch unexpected codes.
You want four groups of customers with about the same number of people in each, based on spend. Which function fits?
Answer: C. pd.qcut(df["spend"], 4)
qcut uses quantiles as bin edges, so each bin holds roughly the same number of rows. cut with a number of bins splits the value range into equal-width intervals, so with skewed spend most customers land in the lowest bin. Pass labels=["Q1", "Q2", "Q3", "Q4"] for readable names.
Each customer should appear once in both tables. How can you make the merge fail loudly if that is not true?
Answer: B. df.merge(other, on="cid", validate="one_to_one")
validate checks the key relationship and raises a MergeError if, for example, a key repeats on either side. Without it, duplicate keys silently multiply rows and inflate totals. "one_to_many" and "many_to_one" are also allowed when only one side must be unique.
A product sold 1 unit at ₹100 and 3 units at ₹200. What is the average price per unit sold?
Answer: B. ₹175
The weighted average is total revenue divided by total units: (100 x 1 + 200 x 3) / 4 = 700 / 4 = 175. A plain mean of the two prices gives 150 and ignores how many units sold at each. In pandas, compute it per group as (price * qty).sum() / qty.sum().
Rows are sorted by user and date. Which line gives the number of days since each user's previous purchase?
Answer: C. df.groupby("user")["date"].diff().dt.days
diff within each user subtracts the previous date of the same user, giving a Timedelta, and .dt.days turns it into a number. Each user's first purchase gets NaN, which is correct. Without the groupby, the first row of one user is compared with the last row of another.
An Indian export writes 3 April 2026 as "03/04/2026". pd.to_datetime turns it into 4 March. What is the safest fix?
Answer: B. pd.to_datetime(s, format="%d/%m/%Y")
Without a hint, pandas reads slash dates month first, the US style. An explicit format makes the day-first order certain and fails loudly if a value does not match; dayfirst=True also works but is a softer hint. Always check dates where the day is 12 or less, because those parse silently either way.
After this line, df is None. Why?
df = df.dropna(inplace=True)
Answer: C. inplace=True changes df and returns None
With inplace=True the method changes the object and returns None, and the assignment then replaces df with that None. Either call df.dropna(inplace=True) without assigning, or, as the pandas docs prefer, write df = df.dropna(). The inplace option gives no real speed gain in most cases.
df[df['notes'].str.contains('refund')] raises an error about masking with NA values. The notes column has some NaN. What fixes it?
Answer: C. Pass na=False to str.contains
str.contains returns NaN for missing values, and a boolean mask with NaN cannot be used to filter. na=False treats missing notes as not matching. Add case=False if 'Refund' and 'REFUND' should match too.
Which pandas call gives the same numbers as SQL RANK() OVER (ORDER BY score DESC)?
Answer: B. df['score'].rank(method='min', ascending=False)
method='min' gives tied rows the lowest rank of the group and then skips, like SQL RANK. 'dense' matches DENSE_RANK and 'first' matches ROW_NUMBER. The default 'average' gives ties values such as 2.5, which SQL does not.
Workers have pandas 2, but one legacy Python task only works with pandas 1.5. Without containers, what is a clean way to run it?
Answer: B. Use @task.virtualenv with pandas==1.5.3
@task.virtualenv (PythonVirtualenvOperator) builds or reuses a separate virtualenv with the listed packages and runs the function there. The rest of the workers keep pandas 2. @task.external_python can point to a pre-built environment to avoid install time.
A Spark job appends a DataFrame with a new column coupon_code to an existing Delta table, without any schema options. What happens?
Answer: D. The write fails because of schema enforcement
Delta enforces the table schema on write, so a mismatched DataFrame is rejected rather than corrupting the table. To add the column on purpose, use mergeSchema or ALTER TABLE. This catches upstream changes early instead of breaking readers later.
A Kafka Streams app reads payments keyed by payment_id and calls groupBy to count by merchant_id. An internal topic ending in -repartition appears. Why?
Answer: D. Records must be reshuffled so each merchant lands in one partition
Aggregation by a new key only works if all records for that key reach the same task. Kafka Streams writes re-keyed records to a repartition topic, which spreads them by merchant_id, then reads them back to aggregate. Repartitioning costs network and storage, so avoid needless key changes.
With adaptive query execution turned off, a groupBy on a small 50 MB DataFrame produces a stage with 200 tasks. Where does 200 come from?
Answer: A. The default of spark.sql.shuffle.partitions
For DataFrame shuffles, Spark uses spark.sql.shuffle.partitions, which is 200 by default. For 50 MB that makes 200 tiny tasks with more overhead than work. Either lower the setting or enable AQE, which merges small shuffle partitions at runtime.
A job updates a DataFrame in a loop 100 times (df = df.join(...).withColumn(...)). Each iteration gets slower and planning takes minutes. What is a standard fix?
Answer: B. Checkpoint the DataFrame every few iterations to cut the lineage
Each loop adds to the logical plan, so the plan and lineage grow without limit and the optimizer slows down. checkpoint() (or localCheckpoint()) saves the data and starts a fresh, short plan. count() runs the work but keeps the long plan.
With the RDD API, why is rdd.reduceByKey(add) usually much better than rdd.groupByKey().mapValues(sum) for word counts?
Answer: A. It combines values on each partition before the shuffle
reduceByKey does a map-side combine: each partition sums its own counts first, so only one value per key per partition is shuffled. groupByKey sends every single value across the network and holds all of a key's values in memory, which can cause out-of-memory errors on hot keys.
A job sorts by event_time, then does groupBy("user_id").agg(F.collect_list("page")). The page lists are sometimes out of time order. Why?
Answer: B. The groupBy shuffle does not keep the earlier sort order
groupBy shuffles rows, and rows from different partitions arrive in no fixed order, so a sort before it is not preserved. Collect structs of (event_time, page), then sort the array with array_sort or sort_array, and extract the page. This gives a deterministic order.
Last month's DataFrame has columns (id, amount). This month's has (id, amount, gst). You want one DataFrame with all three columns and null gst for old rows. What do you use?
Answer: C. old.unionByName(new, allowMissingColumns=True)
unionByName matches columns by name, and allowMissingColumns=True (Spark 3.1+) adds absent columns as null. Plain union matches by position and fails on a different column count; unionByName without the flag also fails. An outer join would merge rows by id instead of stacking them.
You append a DataFrame with a new column discount to an existing Delta table and get a schema mismatch error. What lets the append add the column?
Answer: C. option("mergeSchema", "true")
Delta enforces the table schema on write, which is why the append fails. Setting mergeSchema to true allows compatible schema evolution, such as adding a new nullable column; old rows read it as null. Overwrite would replace data, and inferSchema is a CSV reader option.
df has an int column qty and a float column price. Inside for _, row in df.iterrows(), row["qty"] is 1.0 instead of 1. Why?
Answer: B. Each row becomes one Series with a single dtype
iterrows builds a Series for every row, and a Series has a single dtype, so int and float values share float64. That can break ID lookups and equality checks. itertuples keeps each column's type and is much faster, and a vectorised operation is better still.
What does pd.Series([1, 2, 3, 4]).quantile(0.25) return?
Answer: A. 1.75
pandas uses linear interpolation by default. The position is (n - 1) x 0.25 = 0.75, so the result is three quarters of the way from the 1st value (1) to the 2nd value (2), which is 1.75. Other tools such as Excel's QUARTILE.EXC use different rules, so quartiles can differ slightly between tools.
Both tables have one row where the key k is NaN. What does a.merge(b, on="k") do with them?
Answer: B. It matches the two NaN keys as a pair
Unlike SQL, where NULL never equals NULL, pandas merge treats NaN keys as equal and joins them. This can create false matches, for example many orders with an unknown customer joined to an unknown customer row. Drop or fill missing keys before merging when that is not what you want.
s = pd.Series([np.nan, np.nan]). What are s.sum() and s.mean()?
Answer: C. 0.0 and NaN
pandas skips NaN, and the sum of nothing is defined as 0, while the mean of nothing is undefined, so NaN. A store with no data can therefore show ₹0 revenue instead of "no data". Use s.sum(min_count=1) to get NaN when there are no real values.
What is in df["s"] after this code?
df = pd.DataFrame({"v": [1, 2, 3]}, index=[0, 1, 2])
s = pd.Series([10, 20, 30], index=[1, 2, 3])
df["s"] = sAnswer: D. NaN, 10, 20
pandas aligns the Series to the DataFrame by index label, not by position. Label 0 has no value, labels 1 and 2 get 10 and 20, and label 3 is dropped. This is a common bug after filtering or sorting; use reset_index or .to_numpy() when you really mean position.
You need the most recent AED to INR rate at or before each trade time. Which tool is built for this?
Answer: A. pd.merge_asof(trades, rates, on="time")
merge_asof matches each left row with the nearest key on the right, and its default direction="backward" picks the last one at or before the time. Both inputs must be sorted by the key. An exact merge would only match trades at the very second a rate was published.
df.info() reports 60 MB, but the Python process uses over 600 MB. Most columns hold Python string objects. What shows the true size?
Answer: C. df.memory_usage(deep=True)
By default pandas counts only the 8-byte pointer for each object value, not the strings they point to. deep=True measures the actual string objects, which is often 10 times more. Converting such columns to category or a PyArrow-backed string type usually cuts memory a lot.
A Riyadh office works Sunday to Thursday. Which offset from pd.offsets gives its working days correctly?
Answer: B. CustomBusinessDay(weekmask="Sun Mon Tue Wed Thu")
BDay assumes a Monday to Friday week, which is wrong for Saudi Arabia's Friday-Saturday weekend. CustomBusinessDay takes a weekmask of working days and a separate holidays list of actual dates, such as Eid. The UAE moved to a Saturday-Sunday weekend in 2022, so check each country.
You concat two DataFrames: one has only an int column a, the other only an int column b. What are the result's dtypes?
Answer: A. Both a and b become float64
Each frame lacks the other's column, so concat fills those cells with NaN, and NaN forces int64 to float64. IDs then print as 101.0 and can fail to match in later joins. Use nullable "Int64" columns or check dtypes after combining.
How many of these values fall in the bin (10, 20]?
import pandas as pd vals = [5, 10, 15, 20] pd.cut(vals, bins=[0, 10, 20])
Answer: A. 2
By default pd.cut uses right=True, so each bin includes its right edge and excludes the left: (0, 10] and (10, 20]. 5 and 10 go in the first bin, and 15 and 20 in the second. With right=False, 10 would move to [10, 20) and 20 would fall outside.
a = pd.Series([1.0, np.nan]) and b = pd.Series([1.0, np.nan]). What are (a == b).all() and a.equals(b)?
Answer: B. False and True
With ==, NaN is never equal to anything, including NaN, so the second element compares False and .all() is False. Series.equals treats NaN in the same position as equal and also checks shape and dtype. Use equals, or pandas testing helpers, to compare data in tests.
A rating column has 10 million values from 1 to 5, stored as int64. About how much memory does converting it to int8 save?
Answer: C. About 70 MB
int64 uses 8 bytes per value, so 10 million values take 80 MB. int8 uses 1 byte, about 10 MB, saving about 70 MB. int8 holds −128 to 127, which is plenty for ratings. pd.to_numeric(..., downcast='integer') picks the smallest safe type.
A task pushes a 2 GB pandas DataFrame through Airflow XCom to the next task. What is the better pattern?
Answer: D. Write it to object storage and pass the path in XCom
XCom lives in the metadata database and is meant for small values such as IDs, counts or file paths. Large data belongs in storage like S3 or GCS; tasks exchange references so the scheduler database stays small and fast.
df['income'] has 200 missing values. Why does df[df['income'] == np.nan] return an empty DataFrame?
Answer: A. NaN is never equal to anything, even another NaN, so the comparison is always False; use df["income"].isna()
By the floating-point standard, NaN == NaN is False, so an equality check can never find missing values. pandas provides isna() and notna() for this. The same idea appears in SQL, where you must use IS NULL instead of = NULL.
A DAG file imports tensorflow and pandas at the top and builds a 2,000-task DAG. The UI shows "DagBag import timeout". What is the best fix?
Answer: A. Move heavy imports into the task functions
Parsing a DAG file must finish within dagbag_import_timeout (30 seconds by default). Heavy module-level imports and slow generation logic make parsing slow. Import heavy libraries inside the task callables so they load only on the worker at run time.
A @task loads two 30 GB tables into pandas on an Airflow worker to join them, and the worker keeps getting OOM-killed. What is the best design change?
Answer: B. Run the join in the warehouse or Spark and orchestrate it from Airflow
Airflow is an orchestrator; workers are sized for coordination, not big data processing. Push the join to a system built for it, such as SQL in the warehouse or a Spark job, and let the task just submit and monitor it.
A notebook cached a DataFrame read from a Parquet path. Another job then overwrote that path with corrected data. The notebook keeps showing the old totals. Why?
Answer: B. The cached blocks are reused until you unpersist or refresh
Spark serves the cached plan from cached blocks and does not watch the files for changes. Until you call unpersist(), or REFRESH TABLE / spark.catalog.refreshByPath, you see the old data. Uncached reads of changed files can also fail with file-not-found errors until the metadata is refreshed.
groupBy("customer").agg(F.first("plan")) returns different plans for the same customer on different runs of the same job. Why?
Answer: D. Row order after a shuffle is not fixed, so first() varies
first() returns whichever row the aggregation sees first, and that depends on partition and arrival order, which can change between runs. Spark marks it non-deterministic. Use max_by("plan", "updated_at") or a row_number window to choose by a clear rule.
You change a streaming query from groupBy("city") to groupBy("city", "channel") and restart with the same checkpoint. It fails. What is the right move?
Answer: B. Use a new checkpoint, since the old state no longer fits
The checkpoint holds state shaped by the old aggregation keys, so changing keys, adding stateful operators or changing their order is not compatible. You need a fresh checkpoint and a plan for the gap, such as reprocessing from a chosen offset. Changes like filters or projections on stateless steps are usually allowed.
How many shuffle (Exchange) steps does the physical plan of this code contain, with AQE off and no prior partitioning?
out = (df.repartition(200, "customer_id")
.groupBy("customer_id")
.agg(F.sum("amount")))Answer: B. 1
repartition(200, "customer_id") hash-partitions the data by customer_id, which already satisfies the grouping requirement of groupBy("customer_id"). Spark sees this and skips a second Exchange, so there is one shuffle. Grouping by a different column would add another.
Take a 15-question Pandas MCQ test with a score at the end, or practise the full Pandas question bank and earn points on the leaderboard.