Explain Kafka topics, partitions, offsets and consumer groups to a new teammate.

What the interviewer wants to hear

• A topic is a named, durable log of messages
• Topics are split into partitions for parallelism; order is kept only within a partition
• Messages with the same key go to the same partition
• Offsets mark each message position; consumers commit them to resume
• A consumer group shares partitions so each is read by one member

Model answer

Kafka is a distributed log. A topic is a named stream of messages, such as orders, kept for a retention period on brokers. Each topic is split into partitions, which are the unit of parallelism and replication. Kafka only guarantees order within a partition, so producers set a key, like order_id, and all messages with that key land in the same partition and stay in order. Every message has an offset, its position in the partition. Consumers in a consumer group divide the partitions between them, so each partition is read by one member and the group processes each message once. The group commits offsets, so after a crash or deploy it resumes where it left off, and you can reset offsets to replay. A second consumer group, say for analytics, reads the same data independently with its own offsets.

More easy Data Engineer questions

• Describe a data pipeline you built or would build, from source to dashboard.
• What is the difference between an OLTP database and an OLAP data warehouse?
• Explain a star schema and why it is used in data warehouses.
• What data quality checks would you add to a pipeline?
• Why use an orchestration tool like Airflow instead of cron jobs?
• What is a slowly changing dimension, and how do SCD Types 1, 2 and 3 differ?
• Why would you store analytics data as Parquet rather than CSV or JSON?
• What is change data capture (CDC), and why do data teams use it?
• What does dbt do in a modern data stack, and what are its main features?