Data Engineer
Kafka and stream processing questions
155 questions on Kafka and stream processing for data engineers, easiest first.
Easy 39
- When streaming is neededeasy
- Where Avro fitseasy
- Kafka consumer groupseasy
- What a Kafka offset iseasy
- Topics and partitionseasy
- Job of a Kafka brokereasy
- Replication factor of threeeasy
- Which replica takes the writeseasy
- Producer with acks=0easy
- Records older than retentioneasy
- Turning on producer compressioneasy
- Kafka without ZooKeepereasy
- Total lag over three partitionseasy
- Sending a 5 MB messageeasy
- Why use a schema registryeasy
- What Kafka Connect is foreasy
- Reducing a topic's partitionseasy
- Where group offsets are kepteasy
- Kafka compared with a classic queueeasy
- Event time and processing timeeasy
- Counting events in a one-minute windoweasy
- What a watermark sayseasy
- KStream versus KTableeasy
- Default persistent state storeeasy
- What a Flink checkpoint iseasy
- What keyBy does in Flinkeasy
- How Spark Structured Streaming runseasy
- Kappa architecture in one lineeasy
- Spotting a sequence of eventseasy
- Writing a stream job in SQLeasy
- Encrypting traffic to brokerseasy
- Only one service may writeeasy
- Not running brokers yourselfeasy
- Typos that create topicseasy
- Where to put a trace IDeasy
- A simple background job queueeasy
- Lag for a group with no memberseasy
- Notification or full-state eventeasy
- Adding another consumer groupeasy
Medium 66
- Ordering in Kafkamedium
- Handling late eventsmedium
- Duplicates from at-least-once deliverymedium
- Eight consumers, six partitionsmedium
- Keeping changes to one row in ordermedium
- When to commit Kafka offsetsmedium
- One bad message blocks a streammedium
- What the ISR list containsmedium
- acks=all with one broker downmedium
- What a compacted topic keepsmedium
- Deleting a key from a compacted topicmedium
- Where a keyed record landsmedium
- Records sent with no keymedium
- Duplicates from producer retriesmedium
- Raising linger.msmedium
- Who picks a new leadermedium
- A new group with earliestmedium
- Missing records with latestmedium
- Detecting a crashed consumermedium
- Disk needed for a week of eventsmedium
- Partitions for a throughput targetmedium
- Sending scanned invoices through Kafkamedium
- Adding a field safelymedium
- How Debezium captures changesmedium
- One record that always failsmedium
- Brokers spread over three zonesmedium
- Which offset to commitmedium
- Keeping a year of data cheaplymedium
- Reprocessing since yesterday morningmedium
- Windows that overlapmedium
- Joining two topics in Kafka Streamsmedium
- Joining with a small reference tablemedium
- Matching orders with paymentsmedium
- Restoring state after a crashmedium
- An extra topic appearsmedium
- Threads with nothing to domedium
- Exactly-once in Kafka Streamsmedium
- Reading live counts from a stream appmedium
- Upgrading a stateful Flink jobmedium
- Lag grows behind a slow sinkmedium
- Flink source parallelism of 16medium
- State that grows without limitmedium
- State larger than memorymedium
- Sending rules to every instancemedium
- Restarting a Spark streammedium
- Spark output modesmedium
- Writing to the DB and Kafka togethermedium
- How the outbox pattern worksmedium
- Making a payment consumer safe to replaymedium
- Duplicates that do not mattermedium
- Removing duplicate clicks in a streammedium
- Unique visitors per minute at scalemedium
- Events stamped in the futuremedium
- Under-replicated partitions alertmedium
- Lag climbs after a releasemedium
- One partition far busier than othersmedium
- Key for order lifecycle eventsmedium
- A standby cluster in another regionmedium
- A broker's disk is 95% fullmedium
- One broker leads most partitionsmedium
- When a producer gives upmedium
- Raising fetch.min.bytesmedium
- One client slows the clustermedium
- The target database is downmedium
- Which timestamp a record carriesmedium
- First micro-batch is enormousmedium
Hard 50
- Exactly-once end to endhard
- Which late event is kepthard
- Committing offsets too earlyhard
- Adding partitions to a keyed topichard
- Rows missed by a strict watermarkhard
- Exactly-once inside Kafkahard
- A compacted CDC topichard
- Changing a field type in a topichard
- Adding consumers to cut laghard
- Duplicates that slip past a stream dedupehard
- Consumer down longer than retentionhard
- Two of three replicas losthard
- Risk of unclean leader electionhard
- Old records that outlive retentionhard
- Ordering with five in-flight requestshard
- Slow batch kicks a consumer outhard
- Why cooperative rebalancinghard
- Rolling restarts without rebalanceshard
- The new consumer rebalance protocolhard
- Producers upgrade before consumershard
- Consumers and aborted transactionshard
- Two copies of the same producerhard
- Cutting cross-zone traffic billshard
- Records not yet visible to consumershard
- A late event joins two sessionshard
- When a window first fireshard
- Windows that never closehard
- A late record after the grace periodhard
- Failover takes twenty minuteshard
- One alert per window, not manyhard
- Emitting on a timer with no traffichard
- State lost after a code changehard
- Exactly-once writes from Flink to Kafkahard
- Checkpoints time out under backpressurehard
- Which window Spark has emittedhard
- Duplicate rows from foreachBatchhard
- Memory for hopping window statehard
- Enriching events with a changing tablehard
- Customers get two SMS messageshard
- Cost of salting a hot keyhard
- Failing consumers over to the DR clusterhard
- Disk cap from retention.byteshard
- Rebalancing data without hurting clientshard
- Both compact and deletehard
- Seeing every update on a compacted topichard
- Erasing one person from a long loghard
- What share groups addhard
- startingOffsets changed but ignoredhard
- Stream-stream join running out of memoryhard
- Windowed counts arrive latehard