Integration / Apache Kafka Interview questions

A topic is the named category a stream of records is published to and read from — think of it as a durable, append-only log that producers write to and consumers read from, identified by a string name like orders or payments.

Unlike a queue, a topic doesn't remove a message once it's read; records stay available for every consumer, subject to the topic's configured retention period, so multiple independent consumer groups can each read the same topic from their own position without interfering with one another.

kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Every topic is split into one or more partitions, which is what actually allows a topic to scale beyond what a single broker or a single consumer could handle on its own.

• Producer is an application that sends message/data to the Kafka.
• Consumer is also an application that receives data from Kakfa,
• Broker is nothing but the KAFKA Server,
• Cluster, group of computer nodes sharing workload,
• Topic, Kafka stream
• Partitions, a portion of Topics
• Offset, an unique id for a message within a partition,
• and Consumer Groups, group of consumers acting as a single logical unit.

Topic Name, Partition Number and Offset id identifies a message.

A partition is an ordered, immutable subdivision of a topic's log — each partition is its own independent sequence of records, and a topic with multiple partitions is really multiple parallel logs sharing one topic name.

Ordering is guaranteed within a partition (records are appended in the order they're written and read back in that same order), but there's no guaranteed ordering across different partitions of the same topic. This is the mechanism that gives Kafka horizontal scalability: partitions can be spread across different brokers, and different consumers in a group can each own a different partition, reading and processing in parallel.

kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
# Topic: orders  PartitionCount: 3  ReplicationFactor: 2

A record's partition is chosen either explicitly by the producer, or derived from the record's key via a hashing partitioner, which is what guarantees all records sharing the same key land in the same partition and therefore stay ordered relative to each other.

A producer is the client application that publishes (writes) records to one or more Kafka topics. It's responsible for serializing the record's key and value, deciding (directly or via a partitioner) which partition a record goes to, batching records for efficiency, and handling the broker's acknowledgment according to its configured durability guarantee.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", "order-123", "{\"status\":\"created\"}"));
producer.close();

A key producer setting is acks, which controls how much durability the producer waits for before considering a send successful — acks=0 (fire and forget), acks=1 (leader only), or acks=all (the full in-sync replica set), trading off latency against the risk of data loss on a broker failure.

Apache Kafka is a publish-subscribe open source message broker application.

It is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. It is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant and fast.

Zookeeper builds coordination between different nodes in a cluster. Apache Kafka uses Zookeeper to recover from previously committed offset if any node fails because it works as periodically commit offset.

A consumer is the client application that subscribes to one or more topics and reads records from them, tracking its own position (offset) in each partition it's assigned so it knows what to read next.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "order-processors");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.println(record.value());
    }
}

Unlike many traditional queue consumers, a Kafka consumer pulls data by repeatedly calling poll() rather than having records pushed to it, which lets the consumer control its own processing pace rather than being overwhelmed by an inbound push.

A consumer group is a set of consumers, identified by a shared group.id, that cooperatively read a topic by splitting its partitions among themselves — each partition is consumed by exactly one member of the group at a time, so the group as a whole processes the topic in parallel, but no two consumers in the same group ever read the same partition simultaneously.

group.id=order-processors

This is the mechanism behind Kafka's horizontal consumer scaling: adding more consumers to a group (up to the number of partitions) increases parallel throughput, while different, independently-named consumer groups can each read the exact same topic from their own separate offsets without affecting each other — which is how one topic can simultaneously feed, say, a fraud-detection service and an analytics pipeline, each processing at their own pace.

Apache ZooKeeper is an open-source server which enables highly reliable distributed coordination.

ZooKeeper acts as a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

A broker is a single Kafka server — a process that stores partition data on disk, serves produce and fetch requests from clients, and (in KRaft mode) can also serve as one of the cluster's controller nodes if configured with that role. A Kafka cluster is simply a group of these brokers working together, typically anywhere from a handful to hundreds of nodes depending on scale.

kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Each broker hosts some subset of the cluster's partitions — for a given partition, one broker acts as the leader (handling all reads and writes for it) while other brokers hosting replicas of that same partition act as followers, continuously fetching from the leader to stay in sync. This leader/follower split, repeated across every partition in the cluster and spread across all the brokers, is what balances both storage and request load across the whole cluster rather than concentrating it on one machine.

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

Confluent Platform improves Kafka with additional community and commercial features designed to enhance the streaming experience of both operators and developers in production, at a massive scale.

Kafka’s zero-copy principle is an optimization technique that enables the operating system to transfer data directly from the disk (page cache) to the network socket, bypassing the application (JVM) buffer entirely. This reduces context switches between user/kernel mode and eliminates unnecessary CPU-intensive memory copies, dramatically improving throughput and reducing latency.

Replication means each partition's data is copied to multiple brokers, not just stored on one, so the cluster can survive a broker failure without losing data or availability. The number of copies is set per topic via the replication factor — a replication factor of 3 means every partition has one leader replica and two follower replicas, each living on a different broker.

kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092

Followers continuously fetch new records from their partition's leader to stay caught up; a follower that's fully caught up is considered part of the in-sync replica (ISR) set. If the broker hosting the leader fails, Kafka can promote one of the in-sync followers to be the new leader, so the partition keeps serving reads and writes with no data loss, as long as at least one in-sync replica survived the failure.

KRaft (Kafka Raft) is Kafka's built-in metadata management system, where a dedicated set of controller nodes inside the Kafka cluster itself — using the Raft consensus protocol — store and replicate cluster metadata (which brokers exist, which partitions live where, current leaders) instead of relying on an external system.

As of Apache Kafka 4.0 (released March 2025), KRaft is the only supported mode — the older external metadata-coordination approach has been fully removed from the codebase, not just deprecated, so every current Kafka deployment runs on KRaft.

# server.properties (KRaft mode)
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

In smaller deployments, a broker can also serve as a controller (combined mode); larger clusters typically run a small, dedicated set of controller-only nodes separate from the brokers that serve client traffic, since the controller quorum is now the most operationally critical component in the cluster.

Kafka uses sequential I/O as a primary design choice to achieve its high throughput and performance. By treating data as an append-only log, Kafka can leverage the optimal performance characteristics of both traditional hard disks (HDDs) and Solid State Drives (SSDs), avoiding the significant latency penalties associated with random disk access.

• High Throughput: Sequential writes enable Kafka to handle massive volumes of data (millions of messages per second) with very low latency.
• Cost-Effectiveness: It allows Kafka to use cheaper, high-capacity HDDs effectively, rather than requiring expensive, high-performance storage systems like specialized random-access databases might.
• Simplified Caching Logic: By offloading much of the data management to the OS page cache, Kafka avoids complex in-application caching logic, which can lead to high garbage collection overhead in the Java Virtual Machine (JVM).

Delivery semantics describe the guarantee a consumer gets about whether it might see a message zero, one, or more than once, and Kafka can be configured to support any of the three, depending on how offsets are committed relative to processing.

At-least-once is the most common default in real applications, since it's the simplest to implement correctly and downstream processing can often be made idempotent to tolerate the occasional duplicate; exactly-once requires more setup (transactions, idempotent writes) but removes the need for that downstream deduplication logic entirely.

An offset is a sequential, per-partition number Kafka assigns to each record as it's appended to the log — 0, 1, 2, and so on, within that specific partition. Its main job is tracking consumption progress: a consumer records which offset it has processed up to in each assigned partition, so it knows exactly where to resume if it restarts or a partition gets reassigned to a different consumer during a rebalance.

consumer.commitSync();  // commits the offsets of the most recently polled records

Consumers can commit offsets automatically on an interval (enable.auto.commit=true) or manually via commitSync()/commitAsync() for tighter control over exactly when "processed" is recorded relative to actual work being done — a distinction that directly determines whether an application behaves closer to at-least-once or at-most-once delivery. Committed offsets themselves are stored durably in an internal Kafka topic (__consumer_offsets), not on the consumer's local machine, so progress survives a consumer restarting on a different host entirely.

Kafka exposes five main client-facing APIs, each targeting a different way of interacting with the platform.

Most applications only need the Producer and Consumer APIs directly; Streams and Connect exist specifically to avoid writing that lower-level client code by hand for common patterns like transforming data in-flight or syncing with a database.

A retention policy determines how long Kafka keeps records in a topic before they're eligible for deletion, since a topic's log can't grow forever on finite disk space. It's configured per topic, most commonly by time (retention.ms) or by size (retention.bytes), whichever limit is hit first.

retention.ms=604800000   # 7 days
retention.bytes=-1        # no size-based limit

The purpose is to balance durability against storage cost: a topic feeding real-time processing might only need a few hours or days of retention since consumers are expected to stay caught up, while an event-sourcing or audit-log topic might retain data for months or indefinitely. Retention is independent from whether a message has already been consumed — Kafka doesn't delete a record just because every consumer has read it, so multiple consumer groups can each re-read historical data within the retention window at their own pace.

Every partition has exactly one broker acting as its leader at any given time, and that leader is the sole point of contact for all reads and writes to that partition — producers send records to the leader, and consumers fetch from the leader (unless configured to fetch from a nearby follower for latency reasons in newer Kafka versions).

The leader's other job is coordinating replication: follower brokers hosting replicas of the same partition continuously send fetch requests to the leader to copy new records, and the leader tracks which followers are caught up enough to count as in-sync. If the leader's broker fails, the KRaft controller quorum detects this and triggers a new leader election among the remaining in-sync replicas, so the partition can keep serving traffic without any single broker being a permanent single point of failure for that partition specifically.

A connector is a pre-built, configurable component that moves data between Kafka and an external system, without requiring anyone to write custom producer or consumer code for that integration. Connectors come in two flavors: source connectors pull data from an external system (a database, a log file, an API) into a Kafka topic, and sink connectors push data from a Kafka topic out to an external system (a data warehouse, Elasticsearch, cloud storage).

{
  "name": "jdbc-source-orders",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://db:5432/app",
    "table.whitelist": "orders",
    "topic.prefix": "db-"
  }
}

Connectors run inside a Kafka Connect worker process, which handles the operational concerns — scaling across multiple tasks, tracking source offsets, retrying failures — so the connector configuration itself is usually just a JSON document describing the external system and how to map its data, rather than an application to build and deploy from scratch.

Kafka Streams is a client library for building applications that read from one or more Kafka topics, transform or aggregate that data in-flight, and typically write results back to another Kafka topic — all as a normal JVM application, with no separate stream-processing cluster (like Spark or Flink) to deploy and operate.

StreamsBuilder builder = new StreamsBuilder();
builder.stream("orders")
    .filter((key, value) -> value.contains("high-value"))
    .to("high-value-orders");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Common use cases include filtering and enriching event streams, joining two topics together (e.g. orders joined with customer data), and stateful aggregations (running counts, windowed averages) using local state stores backed by an internal changelog topic for fault tolerance. Because it's just a library rather than a separate processing framework, scaling a Kafka Streams application is as simple as running more instances of the same JVM application, which automatically divide up the input topic's partitions among themselves the same way a Kafka consumer group does.

The kafka-topics.sh script (bundled with the Kafka distribution) manages topics from the command line, with --create being the most common starting point:

kafka-topics.sh --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 2 \
  --bootstrap-server localhost:9092

The three settings worth understanding: --partitions sets the initial parallelism level for the topic (partitions can be increased later, but never decreased), --replication-factor sets how many copies of each partition exist across the cluster (can't exceed the number of brokers), and --bootstrap-server points the CLI at any broker in the target cluster, which it uses to discover the rest of the cluster's metadata. Related commands follow the same pattern: --list shows existing topics, --describe shows a topic's partition/replica detail, and --alter changes settings like partition count on an existing topic.

A serializer converts a Java object (a String, a custom class, an Avro/Protobuf record) into the raw bytes Kafka actually stores and transmits, since Kafka itself is agnostic to message format and just deals in byte arrays. A producer configures separate serializers for the record's key and value, since they're often different types.

key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

Kafka ships built-in serializers for common primitives (StringSerializer, IntegerSerializer, ByteArraySerializer), and structured-format libraries (Avro, Protobuf, JSON Schema) provide their own, often paired with a schema registry so the exact schema used doesn't need to be repeated in every single message. On the consumer side, a matching deserializer reverses the process, converting the raw bytes back into a usable object — a mismatch between a producer's serializer and a consumer's deserializer is one of the most common sources of runtime errors in a Kafka application.

The --list option on kafka-topics.sh returns every topic name in the cluster, which is usually the first command run when exploring an unfamiliar cluster:

kafka-topics.sh --list --bootstrap-server localhost:9092

The output is a plain list of topic names, including Kafka's own internal topics (like __consumer_offsets) unless filtered out. To see detail on a specific topic rather than just its name — partition count, replication factor, leader/ISR per partition — the follow-up command is --describe --topic <name>, which is typically the next step once --list has identified the topic of interest. The Admin API exposes the same functionality programmatically via AdminClient.listTopics() for applications that need to enumerate topics without shelling out to the CLI.

A traditional queue (like classic AMQP/JMS brokers) is built around individual message delivery and removal — a message is typically deleted once acknowledged, and the broker's job is mainly routing and short-term buffering. Kafka is built around a durable, replayable log instead, and that structural difference is what makes it fit high-throughput streaming better.

• Replayability — because messages aren't deleted on consumption, a new consumer can start reading from the beginning of the retention window, or an existing one can reprocess after a bug fix, without the producer resending anything.
• Multiple independent consumers — many consumer groups can read the exact same topic at their own pace, which a queue's single-delivery model doesn't naturally support without fan-out infrastructure on top.
• Partition-based parallelism — a topic scales by adding partitions and consumers, giving high aggregate throughput that a single-queue model struggles to match at the same operational simplicity.
• Batching and efficient sequential disk access — Kafka's storage model is optimized for high-throughput, append-only writes rather than per-message transactional bookkeeping.

The trade-off is that Kafka is a heavier piece of infrastructure to run than a simple queue, and its consumer model (pull-based, offset-tracked) is a different mental model than the push-based, per-message ack/nack pattern many queue systems use — for low-volume task distribution with complex per-message routing rules, a traditional queue can still be the simpler, better fit.

Both move messages between producers and consumers, but they're built around fundamentally different models — Kafka a distributed, partitioned log; RabbitMQ a traditional broker with queues and flexible routing.

The practical rule of thumb: reach for Kafka when the core need is high-throughput event streaming, replayable history, or feeding multiple independent downstream consumers from one source; reach for RabbitMQ when the core need is flexible message routing, work-queue-style task distribution, or simpler per-message delivery guarantees without needing to retain a long history.

These describe what a consumer can be guaranteed about duplicate or missed processing, and the difference comes down entirely to when offsets are committed relative to when the message is actually processed.

Exactly-once in Kafka specifically means exactly-once within Kafka's own read-process-write cycle (using idempotent producers and transactions) — if the processing step also writes to an external system outside Kafka's transactional boundary, that external write still needs its own idempotency handling to preserve the same guarantee end to end.

A small set of dedicated controller nodes (an odd number, commonly 3 or 5, for clean majority voting) forms a Raft consensus group that maintains the cluster's metadata — topic configs, partition assignments, current leaders, ACLs — as a replicated, append-only log, rather than each broker independently tracking its own view.

process.roles=controller
node.id=1
controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093

One controller in the quorum is elected the active controller and is the only one that processes metadata changes at a time; the others replicate its log and stand ready to take over if it fails. Brokers (which may or may not also serve as controllers, depending on cluster size) subscribe to this metadata log and apply changes locally, which is how every broker in the cluster stays in agreement about partition leadership and topic configuration without a central external lookup on every request. This design is what let Kafka 4.0 remove the external coordination system entirely — metadata consensus is now handled natively, inside Kafka's own replicated log mechanism.

acks=all on its own only says the producer waits for acknowledgment from whatever the current in-sync replica (ISR) set happens to be — it doesn't set a floor on how large that set needs to be. If the ISR has shrunk down to just the leader (every follower has fallen behind or failed), acks=all with no further constraint is satisfied by that single replica acknowledging, which offers no more durability than acks=1 in that degraded state.

# topic config
min.insync.replicas=2

# producer config
acks=all

min.insync.replicas sets the actual floor: if the ISR set is smaller than this number when a produce request with acks=all arrives, the broker rejects the write with a NotEnoughReplicasException instead of silently accepting it with weaker durability than intended. The combination — acks=all plus min.insync.replicas=2 on a replication-factor-3 topic, for example — is the standard pattern for guaranteeing a write survives the loss of any single broker, at the cost of the producer getting explicit failures during a broader outage rather than silently accepting weaker guarantees.

When a partition's current leader broker fails or becomes unreachable, the KRaft active controller detects this (via missed heartbeats/session timeout) and selects a new leader from among that partition's in-sync replicas (ISR), then propagates the new leader assignment through the metadata log so every broker and client learns about it.

By default, Kafka only promotes an in-sync replica — one that was fully caught up with the old leader — which guarantees no committed data is lost in the handover. If every replica in the ISR is also unavailable at the same time (a broader outage), the partition simply becomes unavailable for writes rather than silently promoting an out-of-sync replica and risking data loss, unless unclean.leader.election.enable is explicitly turned on to allow that riskier trade-off. Clients (producers and consumers) discover the new leader automatically on their next metadata refresh or when a request to the old leader fails, without needing any manual reconfiguration.

More partitions raise the ceiling on parallel throughput — both how many producers can write concurrently without contention and how many consumers in a group can process in parallel, since consumer parallelism within a group is capped at the partition count. Increasing partitions makes sense when a topic's current consumer group can't keep up because it's already running as many consumer instances as there are partitions, or when a single partition's write throughput is approaching what one broker can realistically sustain.

It's not a free lever, though, and shouldn't be increased casually:

• More partitions means more open file handles and more replication traffic per broker, adding overhead cluster-wide.
• Partition count can only go up, never down, on an existing topic — a topic over-provisioned with partitions can't be shrunk back later without recreating it.
• Increasing partitions on an existing topic changes the key-to-partition mapping going forward, which can break ordering guarantees for keys that previously mapped consistently to one partition.

The practical guidance: size partition count for expected peak parallelism up front where possible, and treat later increases as a deliberate, carefully-considered change rather than a routine scaling knob.

Each consumer in a group periodically sends a heartbeat to the group coordinator (a role held by one of the brokers) to signal it's still alive and processing. If a consumer misses heartbeats for longer than session.timeout.ms, the coordinator considers it dead and removes it from the group, triggering a rebalance that redistributes its assigned partitions among the remaining live consumers.

session.timeout.ms=45000
heartbeat.interval.ms=3000
max.poll.interval.ms=300000

A separate, often-confused setting is max.poll.interval.ms: even if heartbeats (sent on a background thread in modern Kafka clients) keep arriving on time, a consumer that takes too long between calls to poll() — because its processing logic is stuck or too slow — is also considered as having left the group, since Kafka can no longer be sure it's actually making progress on records already handed to it. This is the most common cause of unexpected rebalances in production: slow per-record processing exceeding max.poll.interval.ms, not an actual network or process failure.

A single producer.send() call triggers a multi-step pipeline inside the producer client before any bytes actually reach a broker over the network.

The batching step is deliberate, not incidental — grouping many small records into fewer, larger network requests is a major factor in Kafka's producer throughput, which is why linger.ms and batch.size are two of the most impactful tuning knobs on the producer side.

Producer throughput tuning is mostly about maximizing effective batching and network efficiency without pushing latency higher than the use case can tolerate.

1. Increase batch.size and linger.ms — larger, less frequent batches mean fewer, more efficient network round trips, at the cost of slightly higher per-record latency.
2. Enable compression (compression.type=lz4 or zstd) — smaller batches over the wire mean less network time, often a large win with only modest CPU cost.
3. Increase buffer.memory if producer threads are blocking because the accumulator buffer fills up faster than batches can be sent.
4. Tune acks appropriately for the use case — acks=1 is meaningfully faster than acks=all, and is an acceptable trade-off when occasional data loss on a broker crash is tolerable.
5. Use multiple producer instances or partitions if a single producer's I/O thread is the bottleneck rather than the broker.

The general trade-off running through all of these: throughput optimizations generally cost either latency (larger batches) or durability (weaker acks), so the "optimal" setting depends entirely on which of those the specific application can actually afford to give up.

Consumer lag — the gap between the latest offset produced to a partition and the offset a consumer group has actually processed — is the standard signal that a consumer group isn't keeping up, and the troubleshooting path starts with measuring it precisely before guessing at a cause.

kafka-consumer-groups.sh --describe --group order-processors --bootstrap-server localhost:9092

1. Check if lag is growing or just non-zero — some lag is normal under bursty load; steadily increasing lag over time is the real signal of a problem.
2. Check per-partition lag, not just the group total — lag concentrated on one or two partitions usually points to a hot key or uneven partition assignment, not a group-wide capacity problem.
3. Check processing time per record — slow downstream calls (a database write, an external API) inside the consumer loop are a very common root cause.
4. Check consumer count versus partition count — if the group already has as many consumers as partitions and is still falling behind, only adding more partitions and consumers together will actually add parallelism.
5. Check for frequent rebalances — a group that keeps rebalancing spends time reassigning partitions instead of processing, which itself can look like lag.

The ISR is the set of replicas — leader plus any followers — that are fully caught up with the leader's log at a given moment. It matters because it's the actual pool of candidates eligible to become the new leader if the current one fails, and it's what acks=all and min.insync.replicas are checking against, not the full, nominal replica count.

A topic can have a replication factor of 3 on paper, but if two of those three replicas have fallen behind (say, due to a slow disk or network issue) and dropped out of the ISR, the effective durability at that moment is whatever a single in-sync replica provides — the nominal replication factor overstates the real protection until those followers catch back up and rejoin the ISR. This is exactly why min.insync.replicas exists: it lets a broker refuse writes outright rather than silently accepting them with weaker-than-intended durability during a period when the ISR has shrunk, making the durability gap visible as write failures instead of an invisible risk.

A rebalance is the process of reassigning a topic's partitions among a consumer group's current members, triggered whenever group membership changes — a consumer joins, leaves, crashes, or the subscribed topic's partition count changes.

In the classic (eager) rebalance protocol, every member has to stop processing all its partitions during the whole rebalance, even ones it keeps — a real pause in throughput. The newer cooperative/incremental rebalancing (and the KIP-848 next-generation protocol, GA in Kafka 4.0) narrows this down so only the specific partitions actually being reassigned are paused, letting members keep processing their unaffected partitions throughout, which is a meaningful throughput improvement for large, frequently-changing groups.

Two separate mechanisms combine to deliver exactly-once processing within Kafka's own read-process-write cycle.

Idempotent producer (enable.idempotence=true) assigns each producer a unique ID and a per-partition sequence number to every batch it sends; if a retry occurs (e.g. due to a network timeout after the broker actually received and wrote the batch), the broker recognizes the duplicate sequence number and discards the retry instead of writing the record twice. This alone solves duplicate writes caused by producer-side retries, but doesn't cover multi-partition atomicity or the consume-process-produce pattern.

enable.idempotence=true
transactional.id=order-processor-1

Transactions build on top of idempotence to make a set of writes — potentially spanning multiple partitions or topics, and including a consumer's offset commit — atomic: either all of them become visible to consumers reading with isolation.level=read_committed, or none do. This is what lets a "read from topic A, process, write to topic B, commit offset" application behave as a single atomic unit, so a crash mid-cycle can't leave the offset committed without the corresponding write, or vice versa — the classic source of duplicate or lost processing that exactly-once semantics are meant to eliminate.

Both are ways Kafka reclaims disk space from a topic's log over time, but they remove different things and suit different use cases.

A common real pattern is a topic backing a Kafka Streams state store or serving as a changelog for a cache: compaction guarantees you can always reconstruct the latest state for every key by reading the topic from the start, since old updates to the same key are eventually removed but the newest one for each key persists. A null value acts as a tombstone, marking a key for eventual removal entirely rather than just superseding its prior value.

A custom partitioner overrides Kafka's default key-hash partitioning logic when the built-in behavior doesn't fit a specific routing requirement — for example, routing all records for a given tenant to a dedicated subset of partitions for isolation, rather than letting a generic hash spread them arbitrarily.

public class TenantPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                          Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionCountForTopic(topic);
        String tenantId = (String) key;
        return Math.abs(tenantId.hashCode()) % numPartitions;
    }
    @Override public void configure(Map<String, ?> configs) {}
    @Override public void close() {}
}

partitioner.class=com.example.TenantPartitioner

Custom partitioning is a powerful but sharp tool: it's easy to accidentally create a "hot partition" by routing disproportionately more traffic to one partition than the others, which then becomes a bottleneck regardless of how many total partitions or consumers exist. It's worth reaching for only when the default hash-based distribution genuinely can't express the routing requirement, since uneven partition load is a common and hard-to-diagnose consequence of a poorly-designed custom partitioner.

A state store is a local, embedded key-value store (RocksDB-backed by default, though an in-memory option exists) that a Kafka Streams application uses to hold running state for stateful operations — aggregations, counts, joins — that need to remember something across records rather than processing each one independently.

builder.stream("orders")
    .groupByKey()
    .count(Materialized.as("order-counts-store"));

The critical piece for fault tolerance is that every state store is continuously backed by an internal, compacted changelog topic — every update to the local store is also written to that topic, so if the application instance crashes or is rebalanced to a different machine, the new instance can rebuild the exact same state by replaying the changelog rather than losing that accumulated state entirely. Because the changelog uses compaction, replaying it to rebuild state reads only the latest value per key rather than the full historical update stream, keeping recovery time proportional to the number of distinct keys rather than the total number of updates ever made.

The classic protocol (used by default for years) is eager: every rebalance revokes all partitions from all members first, then reassigns everything from scratch, which means every consumer briefly stops processing, even partitions it ends up keeping. The KIP-848 next-generation protocol, generally available since Kafka 4.0, moves partition assignment logic to the group coordinator (rather than a client-side leader) and only reassigns the specific partitions that actually need to move, letting members keep processing unaffected partitions throughout.

For most current deployments, KIP-848 is the better choice: it meaningfully reduces the processing pause during rebalances, is more resilient to slow or buggy client-side assignment logic since the server now owns that decision, and is where new Kafka development is focused going forward. The classic protocol still has a place in environments running older client libraries not yet updated to support the new protocol, or in scenarios needing a custom client-side assignment strategy that hasn't been ported to the new coordinator-driven model yet.

The practical guidance: default to the new protocol on Kafka 4.0+ clusters with up-to-date clients, and only stay on the classic protocol where a specific compatibility constraint requires it.

A schema registry stores and versions the structured schemas (Avro, Protobuf, JSON Schema) that producers and consumers agree on for a topic's data, so the actual schema doesn't need to be repeated inside every message — a producer registers or looks up the schema once, and messages carry only a compact schema ID reference plus the encoded payload.

# producer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=http://localhost:8081

# consumer
value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
schema.registry.url=http://localhost:8081

On write, the Avro (or Protobuf/JSON Schema) serializer checks the schema registry, registers a new schema version if needed, and prepends the resulting schema ID to the serialized bytes. On read, the deserializer extracts that ID, fetches the matching schema from the registry (caching it locally after the first lookup), and uses it to decode the payload back into a typed object. The real operational value is compatibility enforcement: the registry can be configured to reject a schema change that would break existing consumers (e.g. removing a required field), catching a breaking data-format change at publish time instead of as a runtime deserialization failure discovered by every downstream consumer independently.

Replication in Kafka is pull-based: follower brokers actively fetch new records from the partition's leader, rather than the leader pushing data out to followers, which reuses the exact same fetch-request mechanism consumers use.

The leader tracks, per follower, the highest offset it has confirmed fetching, which is what determines ISR membership — a follower whose fetch position falls too far behind (beyond replica.lag.time.max.ms) is dropped from the ISR until it catches back up. Because followers pull rather than the leader pushing, a slow follower naturally just falls further behind on its own fetch cadence rather than blocking or slowing down the leader's ability to serve producers, which keeps replication lag isolated to the affected follower instead of degrading write latency for everyone.

Tiered storage lets a topic's older log segments move from local broker disks to cheaper, more scalable remote object storage (like S3-compatible storage), while recent, actively-read data stays on fast local disk — extending effective retention far beyond what local broker disk capacity alone could support, without needing ever-larger local volumes.

# broker-level
remote.log.storage.system.enable=true

# per-topic
remote.storage.enable=true
local.retention.ms=86400000     # keep 1 day locally
retention.ms=2592000000          # but retain 30 days total, in the remote tier

From a client's perspective, the split is transparent: a consumer requesting an old offset that's already moved to the remote tier is served by the broker fetching it from remote storage automatically, without the application needing to know or care where the data physically lives. The main trade-off is latency — reads that hit the remote tier are slower than reads served from local disk — which is why local.retention.ms is tuned to keep the actively-consumed recent window on fast local storage while older, rarely-read history moves to the cheaper tier.

Both run inside a Kafka Connect worker and are configured declaratively rather than coded by hand, but they move data in opposite directions.

A single Connect deployment commonly runs both kinds together to build an end-to-end pipeline — for example, a JDBC source connector streaming database changes into a topic, with an Elasticsearch sink connector on the other end indexing those same records for search, with no custom application code written for either half of that pipeline.

A producer can compress an entire batch of records before sending it over the network, using a codec set via compression.type, and that batch stays compressed as it's stored on the broker's disk and replicated to followers — the broker doesn't decompress and recompress it, which keeps compression's benefit intact all the way through storage and replication, not just the initial network hop.

Because compression works on a whole batch rather than record by record, larger batches (via a higher batch.size/linger.ms) generally compress more effectively, which is one more reason producer batching and compression tuning are usually considered together rather than independently. The consumer decompresses transparently based on metadata in the batch header, requiring no consumer-side configuration to match the producer's chosen codec.

Partition assignment strategies decide how a consumer group's partitions get distributed among members during a rebalance, and they differ mainly in how much churn they cause when membership changes.

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Cooperative sticky is the right default choice for most production groups, especially larger ones or ones with frequent membership changes (autoscaling consumer pools, rolling deployments), because it combines minimal partition movement (less state to rebuild, less cache-warming cost per rebalance) with incremental rebalancing (no full-group processing pause). It's less necessary for a small, stable group that rarely changes membership, where the difference between strategies is unlikely to be noticeable in practice.

Kafka security generally layers two separate concerns: authentication (proving who a client is) and authorization (deciding what an authenticated client is allowed to do), typically combined with encryption in transit.

# broker config
listeners=SASL_SSL://0.0.0.0:9093
sasl.enabled.mechanisms=SCRAM-SHA-512
security.inter.broker.protocol=SASL_SSL

SASL (commonly SCRAM or OAUTHBEARER in current deployments, PLAIN for simpler setups) handles authentication, verifying a client's credentials during connection setup; pairing it with SSL/TLS (SASL_SSL) also encrypts the connection itself, so credentials and data aren't sent in the clear.

kafka-acls.sh --add --allow-principal User:order-service \
  --operation Write --topic orders --bootstrap-server localhost:9093

ACLs then handle authorization: once a client's identity is established, ACLs define exactly which operations (read, write, create, describe) that principal is permitted on which resources (specific topics, consumer groups, or the whole cluster). The combination is what lets a shared multi-tenant cluster enforce that, say, a billing service's producer can write to its own topic but has no read or write access to an unrelated HR topic on the same cluster.

unclean.leader.election.enable controls what happens when every in-sync replica of a partition is unavailable at the same time as the leader — a genuinely rare but real failure scenario. When disabled (the default and generally recommended setting), Kafka simply leaves that partition unavailable for writes until an in-sync replica comes back, guaranteeing no committed data is silently lost in the handover.

unclean.leader.election.enable=false

Enabling it lets Kafka instead promote an out-of-sync replica — one that was behind the old leader — to keep the partition available for writes sooner. The cost is real data loss: any records the old leader had that the promoted, out-of-sync replica never received are gone permanently, and worse, new writes to the promoted replica can silently overwrite what would have been the "true" continuation of the log, producing inconsistent history that's hard to detect after the fact. Most production systems value correctness over availability for this specific trade-off and leave it disabled; it's really only defensible for use cases (like some metrics or logging pipelines) where brief availability matters more than occasional, hard-to-notice data loss.

MirrorMaker (MM2, built on the Kafka Connect framework) continuously replicates topics from a source Kafka cluster to a target cluster, typically for disaster recovery, geo-distribution, or feeding a separate analytics cluster without impacting the primary one.

# mm2.properties
clusters=primary, backup
primary.bootstrap.servers=primary-broker:9092
backup.bootstrap.servers=backup-broker:9092

primary->backup.enabled=true
primary->backup.topics=orders,payments

connect-mirror-maker.sh mm2.properties

Because it's Connect-based, MM2 runs as source connectors (one direction of replication per configured flow) and inherits Connect's scaling, offset-tracking, and fault-tolerance behavior rather than being a bespoke standalone tool. Replicated topics on the target cluster are prefixed by default with the source cluster's alias (e.g. primary.orders) to avoid naming collisions and make the data's origin traceable; MM2 also replicates consumer group offsets via offset translation, which is what allows a consumer application to fail over from the primary cluster to the backup and resume roughly where it left off, rather than starting from scratch.

A Kafka Streams application defines a topology — a graph of processing nodes (source, transformations, sink) — and every record pulled from an input topic flows through that graph node by node before any result is produced.

This per-record flow happens continuously as the application polls its input topics, and because the topology is defined once at startup (via the DSL's fluent builder or the lower-level Processor API), Kafka Streams can statically determine which internal topics (repartition, changelog) it needs to create before any data actually starts flowing, rather than discovering that requirement dynamically at runtime.

Kafka brokers, producers, and consumers all expose a large set of runtime metrics via JMX (Java Management Extensions) by default, covering everything from request latency and throughput to under-replicated partition counts and consumer lag — the same interface most JVM applications use for observability, not a Kafka-specific mechanism.

# run with a JMX exporter agent to expose metrics for Prometheus scraping
-javaagent:/opt/jmx_prometheus_javaagent.jar=7071:/opt/kafka-metrics.yml

A few metrics matter especially for day-to-day health checks: UnderReplicatedPartitions (non-zero means some replica has fallen behind and durability is degraded), RequestHandlerAvgIdlePercent (a broker running consistently low signals it's saturated), and consumer-side lag (covered separately via kafka-consumer-groups.sh --describe, since it isn't a simple JMX broker metric). Because raw JMX isn't directly scrapable by tools like Prometheus, most production setups run a JMX-to-Prometheus exporter agent alongside each broker/client JVM, feeding metrics into standard dashboards (Grafana) and alerting pipelines rather than relying on manual JMX console inspection.

Partitions raise the theoretical ceiling on parallelism, but actual throughput depends on more than just that ceiling — several other factors can mean adding partitions delivers little or no real improvement, or even makes things worse.

• Consumer count already below partition count — if a consumer group isn't running enough instances to use the partitions it already has, adding more partitions doesn't help until consumer count also increases.
• Downstream bottleneck, not Kafka — if each consumer's processing is limited by a slow database write or external API call, more partitions just create more parallel callers hitting that same downstream bottleneck.
• Per-broker overhead — every partition adds file handles, replication traffic, and controller metadata load; beyond a certain point, more partitions can degrade controller and broker performance rather than improving client-facing throughput.
• Producer batching efficiency — spreading the same total write volume across many more partitions means smaller batches per partition, which can reduce the batching efficiency that drives producer throughput in the first place.

The practical takeaway: partition count is one input to throughput, not the whole story, and increasing it without also addressing consumer capacity or a genuine downstream bottleneck is a common, ineffective first response to a throughput problem.

Since Apache Kafka 4.0 removed ZooKeeper mode entirely, any cluster still running on it must migrate to KRaft before upgrading to 4.0 or later — there is no direct ZooKeeper-to-4.0 upgrade path. The supported migration runs on a 3.x release that still supports both modes, moving the cluster over while it stays online.

1. Upgrade to a 3.x release that supports migration (Kafka recommends being on a recent 3.x, e.g. 3.9.x, before starting) if not already there.
2. Deploy a new KRaft controller quorum alongside the existing ZooKeeper ensemble, configured in migration mode rather than replacing anything yet.
3. Enable migration on the brokers (zookeeper.metadata.migration.enable=true), which starts syncing existing cluster metadata from ZooKeeper into the new KRaft controllers.
4. Verify the migration has completed and the KRaft controllers hold a full, correct copy of cluster metadata before proceeding further.
5. Roll the brokers one at a time to run in KRaft mode instead of ZooKeeper mode, removing their ZooKeeper dependency broker by broker.
6. Decommission the ZooKeeper ensemble once every broker has been confirmed running purely on KRaft.
7. Only then upgrade to Kafka 4.0+, which requires the cluster to already be fully on KRaft.

The whole process is designed to run without cluster downtime, but it's a multi-step, order-sensitive migration rather than a simple version bump, which is why Kafka's own documentation strongly recommends testing the full sequence against a non-production cluster first.

Traditional Kafka consumption ties partition ownership to a single consumer within a group at a time — if a topic has 3 partitions, at most 3 consumers in a group can process it in parallel, and one slow message at the head of a partition blocks everything behind it for that consumer. Queues for Apache Kafka (KIP-932, early access starting in Kafka 4.0) introduces a share group model that behaves more like a traditional message queue on top of the same underlying log.

The practical difference: share groups trade Kafka's strict per-partition ordering guarantee for queue-like flexibility — multiple workers pulling from the same partition and acknowledging individually — which suits task-distribution workloads where per-message ordering doesn't matter as much as not letting one slow or failing message stall an entire partition's backlog. As an early-access feature in the current 4.x line, it's meant to complement traditional consumer groups for that specific use case, not replace them for ordered event-streaming workloads.