Integration / Apache Kafka Interview questions
Explain the execution flow of a Kafka producer sending a message to a broker?
A single producer.send() call triggers a multi-step pipeline inside the producer client before any bytes actually reach a broker over the network.
flowchart TD
A[producer.send called with a ProducerRecord] --> B[Key and value serialized to bytes]
B --> C{Partition explicitly specified?}
C -- No --> D[Partitioner selects a partition, e.g. hash of the key]
C -- Yes --> E[Use the specified partition]
D --> F[Record appended to an in-memory batch for that partition]
E --> F
F --> G{Batch full or linger.ms elapsed?}
G -- No --> H[Wait, accumulating more records in the batch]
G -- Yes --> I[Batch sent to the partition's current leader broker]
I --> J{acks setting satisfied by broker response?}
J -- Yes --> K[send() future completes successfully]
J -- No / timeout --> L[Retry, up to configured retry limit, then fail the future]
The batching step is deliberate, not incidental — grouping many small records into fewer, larger network requests is a major factor in Kafka's producer throughput, which is why linger.ms and batch.size are two of the most impactful tuning knobs on the producer side.
What determines when an accumulated batch of records is actually sent to the broker?
What happens if the broker's response doesn't satisfy the configured acks setting?
More Related questions...
