Integration / Apache Kafka Interview questions
1. What is a Kafka topic?
A topic is the named category a stream of records is published to and read from — think of it as a durable, append-only log that producers write to and consumers read from, identified by a string name like orders or payments . Unlike a queue, a topic doesn't remove a message once it's read;...
2. Mention some of the Apache Kafka terminologies.
Producer is an application that sends message/data to the Kafka. Consumer is also an application that receives data from Kakfa, Broke r is nothing but the KAFKA Server, Cluster , group of computer nodes sharing workload, Topic , Kafka stream Partition s, a portion of Topics Offset , an unique id ...
3. What is a partition in Kafka?
A partition is an ordered, immutable subdivision of a topic's log — each partition is its own independent sequence of records, and a topic with multiple partitions is really multiple parallel logs sharing one topic name. Ordering is guaranteed within a partition (records are appended in the...
4. What is the Global Unique identifier of a Kafka Message?
Topic Name, Partition Number and Offset id identifies a message.
5. What is Apache Kafka?
Apache Kafka is a publish-subscribe open source message broker application. It is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. It is used for building real-time data pipelines and streaming apps. It is horizontally scal...
6. What is a Kafka producer?
A producer is the client application that publishes (writes) records to one or more Kafka topics. It's responsible for serializing the record's key and value, deciding (directly or via a partitioner) which partition a record goes to, batching records for efficiency, and handling the broker's ackn...
7. Explain the role of the ZooKeeper in Kafka.
Zookeeper builds coordination between different nodes in a cluster. Apache Kafka uses Zookeeper to recover from previously committed offset if any node fails because it works as periodically commit offset.
8. What is a Kafka consumer?
A consumer is the client application that subscribes to one or more topics and reads records from them, tracking its own position (offset) in each partition it's assigned so it knows what to read next. Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.pu...
9. What is Apache ZooKeeper?
Apache ZooKeeper is an open-source server which enables highly reliable distributed coordination. ZooKeeper acts as a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
10. What is a consumer group in Kafka?
A consumer group is a set of consumers, identified by a shared group.id , that cooperatively read a topic by splitting its partitions among themselves — each partition is consumed by exactly one member of the group at a time, so the group as a whole processes the topic in parallel, but no t...
11. Difference between Apache Kafka and Confluent Kafka.
Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open-sourced by LinkedIn in 2011, Kafka has quickly evolv...
12. What is a Kafka broker?
A broker is a single Kafka server — a process that stores partition data on disk, serves produce and fetch requests from clients, and (in KRaft mode) can also serve as one of the cluster's controller nodes if configured with that role. A Kafka cluster is simply a group of these brokers work...
13. Define replication in Kafka?
Replication means each partition's data is copied to multiple brokers, not just stored on one, so the cluster can survive a broker failure without losing data or availability. The number of copies is set per topic via the replication factor — a replication factor of 3 means every partition ...
14. Kafka's zero-copy principle.
Kafkas zero-copy principle is an optimization technique that enables the operating system to transfer data directly from the disk (page cache) to the network socket, bypassing the application (JVM) buffer entirely. This reduces context switches between user/kernel mode and eliminates unnecessary...
15. What is KRaft mode in Kafka?
KRaft (Kafka Raft) is Kafka's built-in metadata management system, where a dedicated set of controller nodes inside the Kafka cluster itself — using the Raft consensus protocol — store and replicate cluster metadata (which brokers exist, which partitions live where, current leaders) i...
16. Explain sequential I/O principle in Kafka.
Kafka uses sequential I/O as a primary design choice to achieve its high throughput and performance. By treating data as an append-only log, Kafka can leverage the optimal performance characteristics of both traditional hard disks (HDDs) and Solid State Drives (SSDs), avoiding the significant lat...
17. What are the types of message delivery semantics in Kafka?
Delivery semantics describe the guarantee a consumer gets about whether it might see a message zero, one, or more than once, and Kafka can be configured to support any of the three, depending on how offsets are committed relative to processing. Semantic Guarantee At-most-once A message may be los...
18. What is a Kafka offset used for?
An offset is a sequential, per-partition number Kafka assigns to each record as it's appended to the log — 0, 1, 2, and so on, within that specific partition. Its main job is tracking consumption progress : a consumer records which offset it has processed up to in each assigned partition, s...
19. List the core APIs provided by Kafka?
Kafka exposes five main client-facing APIs, each targeting a different way of interacting with the platform. API Purpose Producer API Publish records to topics. Consumer API Subscribe to and read records from topics. Streams API Build stream-processing applications (filter, join, aggregate) direc...
20. What is the purpose of a Kafka topic's retention policy?
A retention policy determines how long Kafka keeps records in a topic before they're eligible for deletion, since a topic's log can't grow forever on finite disk space. It's configured per topic, most commonly by time ( retention.ms ) or by size ( retention.bytes ), whichever limit is hit first. ...
21. Describe the role of a partition leader in Kafka?
Every partition has exactly one broker acting as its leader at any given time, and that leader is the sole point of contact for all reads and writes to that partition — producers send records to the leader, and consumers fetch from the leader (unless configured to fetch from a nearby follow...
22. What is a Kafka Connect connector?
A connector is a pre-built, configurable component that moves data between Kafka and an external system, without requiring anyone to write custom producer or consumer code for that integration. Connectors come in two flavors: source connectors pull data from an external system (a database, a log ...
23. What are Kafka Streams used for?
Kafka Streams is a client library for building applications that read from one or more Kafka topics, transform or aggregate that data in-flight, and typically write results back to another Kafka topic — all as a normal JVM application, with no separate stream-processing cluster (like Spark ...
24. How do you create a topic using the Kafka CLI?
The kafka-topics.sh script (bundled with the Kafka distribution) manages topics from the command line, with --create being the most common starting point: kafka-topics.sh --create \ --topic orders \ --partitions 3 \ --replication-factor 2 \ --bootstrap-server localhost:9092 The three settings wor...
25. What is a serializer in Kafka producer configuration?
A serializer converts a Java object (a String, a custom class, an Avro/Protobuf record) into the raw bytes Kafka actually stores and transmits, since Kafka itself is agnostic to message format and just deals in byte arrays. A producer configures separate serializers for the record's key and value...
26. How do you list existing topics in a Kafka cluster?
The --list option on kafka-topics.sh returns every topic name in the cluster, which is usually the first command run when exploring an unfamiliar cluster: kafka-topics.sh --list --bootstrap-server localhost:9092 The output is a plain list of topic names, including Kafka's own internal topics (lik...
27. Why is Kafka better suited than a traditional message queue for high-throughput event streaming?
A traditional queue (like classic AMQP/JMS brokers) is built around individual message delivery and removal — a message is typically deleted once acknowledged, and the broker's job is mainly routing and short-term buffering. Kafka is built around a durable, replayable log instead, and that ...
28. How does Kafka differ from RabbitMQ?
Both move messages between producers and consumers, but they're built around fundamentally different models — Kafka a distributed, partitioned log; RabbitMQ a traditional broker with queues and flexible routing. Kafka RabbitMQ Records persist in a log for a configured retention period, repl...
29. What is the difference between at-least-once, at-most-once, and exactly-once delivery semantics?
These describe what a consumer can be guaranteed about duplicate or missed processing, and the difference comes down entirely to when offsets are committed relative to when the message is actually processed. At-most-once At-least-once Exactly-once Commit offset BEFORE processing. Commit offset AF...
30. How does Kafka's KRaft controller quorum manage cluster metadata?
A small set of dedicated controller nodes (an odd number, commonly 3 or 5, for clean majority voting) forms a Raft consensus group that maintains the cluster's metadata — topic configs, partition assignments, current leaders, ACLs — as a replicated, append-only log, rather than each b...
31. Why should you configure min.insync.replicas alongside acks=all?
acks=all on its own only says the producer waits for acknowledgment from whatever the current in-sync replica (ISR) set happens to be — it doesn't set a floor on how large that set needs to be. If the ISR has shrunk down to just the leader (every follower has fallen behind or failed), acks=...
32. How does Kafka handle partition leader election?
When a partition's current leader broker fails or becomes unreachable, the KRaft active controller detects this (via missed heartbeats/session timeout) and selects a new leader from among that partition's in-sync replicas (ISR), then propagates the new leader assignment through the metadata log s...
33. When should you increase the number of partitions for a topic?
More partitions raise the ceiling on parallel throughput — both how many producers can write concurrently without contention and how many consumers in a group can process in parallel, since consumer parallelism within a group is capped at the partition count. Increasing partitions makes sen...
34. What happens when a consumer in a group fails to send a heartbeat in time?
Each consumer in a group periodically sends a heartbeat to the group coordinator (a role held by one of the brokers) to signal it's still alive and processing. If a consumer misses heartbeats for longer than session.timeout.ms , the coordinator considers it dead and removes it from the group, tri...
35. Explain the execution flow of a Kafka producer sending a message to a broker?
A single producer.send() call triggers a multi-step pipeline inside the producer client before any bytes actually reach a broker over the network. flowchart TD A[producer.send called with a ProducerRecord] --> B[Key and value serialized to bytes] B --> C{Partition explicitly specified?} C -- No -...
36. How can you optimize Kafka producer throughput?
Producer throughput tuning is mostly about maximizing effective batching and network efficiency without pushing latency higher than the use case can tolerate. Increase batch.size and linger.ms — larger, less frequent batches mean fewer, more efficient network round trips, at the cost of sli...
37. How do you troubleshoot consumer lag in a Kafka application?
Consumer lag — the gap between the latest offset produced to a partition and the offset a consumer group has actually processed — is the standard signal that a consumer group isn't keeping up, and the troubleshooting path starts with measuring it precisely before guessing at a cause. ...
38. Why is the in-sync replica (ISR) set important for durability?
The ISR is the set of replicas — leader plus any followers — that are fully caught up with the leader's log at a given moment. It matters because it's the actual pool of candidates eligible to become the new leader if the current one fails, and it's what acks=all and min.insync.replic...
39. Explain the lifecycle of a Kafka consumer group rebalance?
A rebalance is the process of reassigning a topic's partitions among a consumer group's current members, triggered whenever group membership changes — a consumer joins, leaves, crashes, or the subscribed topic's partition count changes. flowchart TD A[Trigger: consumer joins/leaves, crash, ...
40. How does Kafka achieve exactly-once semantics with idempotent producers and transactions?
Two separate mechanisms combine to deliver exactly-once processing within Kafka's own read-process-write cycle. Idempotent producer ( enable.idempotence=true ) assigns each producer a unique ID and a per-partition sequence number to every batch it sends; if a retry occurs (e.g. due to a network t...
41. What is the difference between log compaction and log deletion cleanup policies?
Both are ways Kafka reclaims disk space from a topic's log over time, but they remove different things and suit different use cases. Deletion (cleanup.policy=delete) Compaction (cleanup.policy=compact) Removes entire old segments once they exceed retention.ms/retention.bytes. Removes older record...
42. How do you implement a custom partitioner in Kafka?
A custom partitioner overrides Kafka's default key-hash partitioning logic when the built-in behavior doesn't fit a specific routing requirement — for example, routing all records for a given tenant to a dedicated subset of partitions for isolation, rather than letting a generic hash spread...
43. How does Kafka Streams manage local state using state stores?
A state store is a local, embedded key-value store (RocksDB-backed by default, though an in-memory option exists) that a Kafka Streams application uses to hold running state for stateful operations — aggregations, counts, joins — that need to remember something across records rather t...
44. Which is better and why: the classic consumer rebalance protocol or the new KIP-848 protocol?
The classic protocol (used by default for years) is eager : every rebalance revokes all partitions from all members first, then reassigns everything from scratch, which means every consumer briefly stops processing, even partitions it ends up keeping. The KIP-848 next-generation protocol, general...
45. How do you integrate a schema registry with Kafka producers and consumers?
A schema registry stores and versions the structured schemas (Avro, Protobuf, JSON Schema) that producers and consumers agree on for a topic's data, so the actual schema doesn't need to be repeated inside every message — a producer registers or looks up the schema once, and messages carry o...
46. Explain the internal working of Kafka's replication protocol between leader and follower brokers?
Replication in Kafka is pull-based : follower brokers actively fetch new records from the partition's leader, rather than the leader pushing data out to followers, which reuses the exact same fetch-request mechanism consumers use. sequenceDiagram participant P as Producer participant L as Leader ...
47. How do you configure tiered storage for a Kafka topic?
Tiered storage lets a topic's older log segments move from local broker disks to cheaper, more scalable remote object storage (like S3-compatible storage), while recent, actively-read data stays on fast local disk — extending effective retention far beyond what local broker disk capacity al...
48. What is the difference between Kafka Connect source and sink connectors?
Both run inside a Kafka Connect worker and are configured declaratively rather than coded by hand, but they move data in opposite directions. Source Connector Sink Connector Pulls data FROM an external system INTO a Kafka topic. Pulls data FROM a Kafka topic and pushes it INTO an external system....
49. How does Kafka support message compression?
A producer can compress an entire batch of records before sending it over the network, using a codec set via compression.type , and that batch stays compressed as it's stored on the broker's disk and replicated to followers — the broker doesn't decompress and recompress it, which keeps comp...
50. When would you choose the cooperative sticky partition assignment strategy?
Partition assignment strategies decide how a consumer group's partitions get distributed among members during a rebalance, and they differ mainly in how much churn they cause when membership changes. Strategy Behavior Range Assigns contiguous partition ranges per topic; can be uneven with multipl...
51. How do you secure a Kafka cluster with SASL and ACLs?
Kafka security generally layers two separate concerns: authentication (proving who a client is) and authorization (deciding what an authenticated client is allowed to do), typically combined with encryption in transit. # broker config listeners=SASL_SSL://0.0.0.0:9093 sasl.enabled.mechanisms=SCRA...
52. Why should unclean leader election be disabled in most production clusters?
unclean.leader.election.enable controls what happens when every in-sync replica of a partition is unavailable at the same time as the leader — a genuinely rare but real failure scenario. When disabled (the default and generally recommended setting), Kafka simply leaves that partition unavai...
53. How do you configure MirrorMaker for cross-cluster replication?
MirrorMaker (MM2, built on the Kafka Connect framework) continuously replicates topics from a source Kafka cluster to a target cluster, typically for disaster recovery, geo-distribution, or feeding a separate analytics cluster without impacting the primary one. # mm2.properties clusters=primary, ...
54. Explain the execution flow of a Kafka Streams topology processing a record?
A Kafka Streams application defines a topology — a graph of processing nodes (source, transformations, sink) — and every record pulled from an input topic flows through that graph node by node before any result is produced. flowchart TD A[Record polled from source topic partition] -->...
55. How does Kafka report and expose broker and consumer metrics for monitoring?
Kafka brokers, producers, and consumers all expose a large set of runtime metrics via JMX (Java Management Extensions) by default, covering everything from request latency and throughput to under-replicated partition counts and consumer lag — the same interface most JVM applications use for...
56. Why doesn't increasing partition count always improve throughput?
Partitions raise the theoretical ceiling on parallelism, but actual throughput depends on more than just that ceiling — several other factors can mean adding partitions delivers little or no real improvement, or even makes things worse. Consumer count already below partition count — i...
57. How do you migrate a Kafka cluster from ZooKeeper mode to KRaft mode?
Since Apache Kafka 4.0 removed ZooKeeper mode entirely, any cluster still running on it must migrate to KRaft before upgrading to 4.0 or later — there is no direct ZooKeeper-to-4.0 upgrade path. The supported migration runs on a 3.x release that still supports both modes, moving the cluster...
58. What is the difference between Kafka's Queues feature (KIP-932) and traditional partitioned consumption?
Traditional Kafka consumption ties partition ownership to a single consumer within a group at a time — if a topic has 3 partitions, at most 3 consumers in a group can process it in parallel, and one slow message at the head of a partition blocks everything behind it for that consumer. Queue...
