41. Account events processed out of order
An "account debited" event is sometimes processed before the "account opened" event for the same account. The producer sends messages without a key to a 12-partition topic. What is the fix?
- A.Create a separate consumer group for each account
- B.Increase the number of partitions to 48
- C.Have each consumer buffer messages for a few seconds and sort them by producer timestamp before processing
- D.Use the account id as the message key so all of an account's events go to the same partition
Show answer
Answer: D. Use the account id as the message key so all of an account's events go to the same partition
Kafka guarantees order only within a partition. Without a key, events for one account spread across partitions and are consumed in parallel. Keying by account id sends them to one partition, in order, while different accounts still scale across partitions. Sorting by timestamp in the consumer cannot fix events that arrive in different partitions.
42. Calculating consumer lag
A Kafka partition's log-end offset is 10,500. The consumer group's committed offset is 10,200 and the consumer processes 50 records per second while producers write 20 per second. What is the lag, and how long to catch up?
- A.300 records, about 6 seconds
- B.10,200 records, about 3 minutes
- C.300 records, about 10 seconds
- D.10,500 records, about 3.5 minutes
Show answer
Answer: C. 300 records, about 10 seconds
Lag is log-end offset minus committed offset: 10,500 - 10,200 = 300. The consumer gains on the producer at 50 - 20 = 30 records per second, so 300 / 30 = 10 seconds. Dividing by 50 alone ignores the records still being produced.
43. Duplicates from producer retries
A Kafka producer times out waiting for an ack and retries, and the topic ends up with duplicate records. What setting addresses this?
- A.enable.idempotence=true on the producer
- B.acks=0 so the producer never waits
- C.auto.offset.reset=earliest on consumers
- D.A larger linger.ms to batch records
Show answer
Answer: A. enable.idempotence=true on the producer
With idempotence the producer gets an ID and sends sequence numbers, so the broker discards a retried batch it already wrote. It is on by default in modern Kafka clients unless conflicting settings disable it. acks=0 hides the problem by risking data loss instead.
44. Adding partitions to a keyed topic
A keyed Kafka topic is increased from 6 to 12 partitions to handle more load. What risk does this create for per-key ordering?
- A.Kafka rewrites old records into the new partitions
- B.Consumers must be restarted with new group IDs
- C.Each key will now be split across 12 partitions
- D.Keys can map to a new partition while old events wait
Show answer
Answer: D. Keys can map to a new partition while old events wait
The default partitioner is hash(key) mod partition count, so changing the count moves many keys to different partitions. New events for a key can be read from the new partition before older ones in the old partition are consumed. Plan partition counts ahead, or drain and switch carefully.
45. Streaming counts that run high
A streaming job counts purchases per user for a real-time feature. After a consumer restart, some counts are higher than the source data. What is the cause and fix?
- A.Clock skew; switch every server to UTC
- B.The model needs retraining
- C.The feature store is too slow; add caching
- D.Messages were redelivered under at-least-once delivery; deduplicate by event ID or make updates idempotent
Show answer
Answer: D. Messages were redelivered under at-least-once delivery; deduplicate by event ID or make updates idempotent
Most streaming systems guarantee at-least-once delivery, so after a failure some events are processed twice. Counters that just increment then double count. Use event IDs for deduplication, idempotent upserts or exactly-once features of the stream processor, and reconcile against batch counts.
46. Late events missing from counts
A Flink job computes 10-minute counts with a watermark 2 minutes behind the latest event time. Mobile events often arrive 5 minutes late. What happens to them?
- A.They are added to the next window automatically instead
- B.They make the streaming job crash with an error
- C.Their window has closed, so they are dropped or side-output
- D.They are counted twice so that nothing is lost
Show answer
Answer: C. Their window has closed, so they are dropped or side-output
The watermark tells the engine when a window is complete. Events with timestamps older than the watermark are late; unless you set allowed lateness or a side output, they are discarded, and features undercount.
47. Showing JSON while it streams
You stream a structured JSON reply to a UI to show fields as they arrive. JSON.parse fails on every chunk until the end. What is the right approach?
- A.Use an incremental JSON parser that accepts partial input
- B.Retry the stream until the first chunk is valid JSON
- C.Set temperature to 0 so each chunk is complete JSON
- D.Ask the model to send the closing brace first
Show answer
Answer: A. Use an incremental JSON parser that accepts partial input
A partial stream is not valid JSON, so normal parsers fail. Incremental or partial JSON parsers (or SDK helpers that emit partial objects) let you render fields as they appear, while you still validate the final full object.
48. Adding partitions to a keyed topic
A topic keyed by order_id has 12 partitions, and consumers rely on all events for an order being in order. You increase it to 24 partitions. What is the risk?
- A.Old messages are deleted
- B.New events for an existing order_id may go to a different partition than its earlier events, so per-key ordering can break during the transition
- C.Kafka rejects keyed messages after the change
- D.Consumers can no longer use consumer groups
Show answer
Answer: B. New events for an existing order_id may go to a different partition than its earlier events, so per-key ordering can break during the transition
The default partitioner maps a key with hash(key) mod number of partitions. Changing the count changes the mapping, so a key that used to land in partition 3 may now land elsewhere while old events are still being consumed from partition 3. Plan partition counts ahead, or create a new topic and migrate consumers carefully.
49. Rows missed by a strict watermark
An incremental job loads WHERE updated_at > MAX(updated_at) in the target. Replication from the source sometimes commits rows a few minutes late with older updated_at values, and those rows never appear. What is a robust fix?
- A.Run the job less often
- B.Re-read a lookback window (for example updated_at > max - 3 hours) and MERGE on the primary key, so overlap is safe
- C.Use updated_at >= instead of >
- D.Switch from MERGE to INSERT for speed
Show answer
Answer: B. Re-read a lookback window (for example updated_at > max - 3 hours) and MERGE on the primary key, so overlap is safe
A strict watermark assumes rows arrive in updated_at order, which replication lag and long transactions break. Reprocessing an overlapping window catches late commits, and an idempotent MERGE on the key prevents duplicates from the overlap. Log-based CDC avoids the problem by reading commit order directly.
50. Adding consumers to cut lag
A Kafka topic has 12 partitions and consumer lag keeps growing. The team scales the consumer group from 12 to 20 instances. What happens?
- A.Throughput rises by about two thirds
- B.Each partition is shared by about 1.7 consumers
- C.The topic gains 8 new partitions
- D.Only 12 consumers get partitions; 8 sit idle
Show answer
Answer: D. Only 12 consumers get partitions; 8 sit idle
Within a consumer group, each partition is read by one consumer at a time, so parallelism is capped by the partition count. To go wider, add partitions (which changes key-to-partition mapping) or make each consumer faster, for example with batching.
51. Risk of unclean leader election
All in-sync replicas of a partition are down, and only an out-of-sync follower is alive. If unclean.leader.election.enable=true, what is the trade-off?
- A.It comes back online sooner, but some committed records may be lost
- B.The partition stays offline until a clean replica returns
- C.Producers must switch to acks=0 until the ISR recovers
- D.Consumers reread the partition from offset zero
Show answer
Answer: A. It comes back online sooner, but some committed records may be lost
Unclean leader election lets a replica that was not in sync become leader. The partition becomes available again, but any records the old leader had committed that this follower never copied are gone. The default is false, which keeps data safe at the cost of the partition staying unavailable.
52. Why cooperative rebalancing
With the classic eager protocol, every rebalance makes all consumers drop all partitions. What does the CooperativeStickyAssignor change?
- A.It removes the need for a group coordinator
- B.It assigns every partition to every consumer
- C.It moves partitions only at night to avoid busy hours
- D.Only partitions changing owner are paused; others keep running
Show answer
Answer: D. Only partitions changing owner are paused; others keep running
Eager rebalancing is stop-the-world: everyone revokes everything, then gets a new assignment. Cooperative incremental rebalancing revokes only the partitions that change owner, so most consumers keep processing. Sticky assignment also tries to keep partitions where they were, which preserves local caches and state.
53. Producers upgrade before consumers
Producers will move to a new schema before consumers are updated, so old consumers must read new data. Which compatibility mode protects this rollout order?
- A.FORWARD
- B.BACKWARD
- C.NONE
- D.It does not matter, Avro handles any change
Show answer
Answer: A. FORWARD
FORWARD compatibility means data written with the new schema can be read by the previous schema. That is what you need when producers upgrade first. BACKWARD suits the opposite order, where consumers upgrade first. FULL checks both ways.
54. Consumers and aborted transactions
A transactional producer writes records and then aborts the transaction. A consumer uses isolation.level=read_committed. What does it see?
- A.It does not see the aborted records
- B.It sees them with a header marking them aborted
- C.It sees them once the producer retries
- D.It sees them only if it uses acks=all
Show answer
Answer: A. It does not see the aborted records
With read_committed, the consumer only returns records from committed transactions and skips aborted ones. It also stops at the last stable offset, so it waits behind open transactions. The default read_uncommitted returns everything, including aborted records.
55. Two copies of the same producer
A stream app instance freezes, a replacement starts with the same transactional.id, and then the old one wakes up and tries to commit. What does Kafka do?
- A.It accepts both commits, creating duplicates
- B.It fences the old producer, so its writes fail
- C.It merges the two transactions into one
- D.It lets whichever commits first win, then aborts the other
Show answer
Answer: B. It fences the old producer, so its writes fail
When the new instance calls initTransactions with the same transactional.id, the coordinator bumps the producer epoch. The old zombie still has the old epoch, so its writes and commit are rejected with a fencing error. This is how Kafka prevents duplicate output from a revived instance.
56. Which window Spark has emitted
A Spark query uses withWatermark("ts", "10 minutes"), 5-minute tumbling windows and append output mode. The highest event time seen so far is 12:20. Which window has been written to the sink?
- A.[12:15, 12:20)
- B.[12:10, 12:15)
- C.[12:05, 12:10)
- D.No window, since append mode waits for the query to stop
Show answer
Answer: C. [12:05, 12:10)
The watermark is 12:20 − 10 minutes = 12:10. In append mode, a window is written only once the watermark passes its end, so [12:05, 12:10) is final and emitted. [12:10, 12:15) could still get late data, so it waits.
57. Duplicate rows from foreachBatch
A Spark query uses foreachBatch to append each micro-batch into a warehouse table. After a crash, one batch appears twice. What is the standard fix?
- A.Make the write idempotent using the batchId, e.g. a MERGE
- B.Switch the trigger to run once per hour instead
- C.Remove the checkpointLocation
- D.Set the Kafka source to startingOffsets=latest
Show answer
Answer: A. Make the write idempotent using the batchId, e.g. a MERGE
Spark may re-run the last batch after a failure, so foreachBatch gives at-least-once by default. Each call gets a batchId that is the same on replay. Recording processed batch IDs in the target, or using a MERGE on a unique key, makes the replay harmless.
58. Memory for hopping window state
A job tracks 2 million active users in 1-hour hopping windows that advance every 10 minutes. Each window aggregate takes about 100 bytes. Roughly how much state is held at once?
- A.About 200 MB
- B.About 12 GB
- C.About 20 MB
- D.About 1.2 GB
Show answer
Answer: D. About 1.2 GB
Each user is in 60 ÷ 10 = 6 open windows at a time. 2,000,000 × 6 × 100 bytes = 1.2 billion bytes, about 1.2 GB, before store overhead and grace periods. Overlapping windows multiply state, which is why window design affects cost.
59. Cost of salting a hot key
To spread the big merchant's load, you change its key to merchant_id + '-' + (order_id % 8). What do you give up?
- A.Nothing; ordering is still kept for every order
- B.One ordered stream for all of that merchant's events
- C.Replication for all of that merchant's records
- D.The ability to compress those records
Show answer
Answer: B. One ordered stream for all of that merchant's events
Salting spreads one key over several partitions, so throughput improves, but events for that merchant no longer have one order. Events for a single order still share a partition, since order_id picks the salt. Any per-merchant aggregate now needs a second step to combine the 8 parts.
60. Disk cap from retention.bytes
A topic has 24 partitions, replication factor 3 and retention.bytes = 10 GB. Ignoring the active segment, what is the most the topic can use across the cluster?
- A.10 GB
- B.30 GB
- C.720 GB
- D.240 GB
Show answer
Answer: C. 720 GB
retention.bytes applies to each partition, not the whole topic. 24 partitions × 10 GB = 240 GB for one copy, and each replica keeps its own, so × 3 = 720 GB. Many teams set this expecting a topic-wide cap and are surprised.
