Integration / Apache Kafka Interview questions
How do you implement a custom partitioner in Kafka?
A custom partitioner overrides Kafka's default key-hash partitioning logic when the built-in behavior doesn't fit a specific routing requirement — for example, routing all records for a given tenant to a dedicated subset of partitions for isolation, rather than letting a generic hash spread them arbitrarily.
public class TenantPartitioner implements Partitioner { @Override public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { int numPartitions = cluster.partitionCountForTopic(topic); String tenantId = (String) key; return Math.abs(tenantId.hashCode()) % numPartitions; } @Override public void configure(Map<String, ?> configs) {} @Override public void close() {} }
partitioner.class=com.example.TenantPartitioner
Custom partitioning is a powerful but sharp tool: it's easy to accidentally create a "hot partition" by routing disproportionately more traffic to one partition than the others, which then becomes a bottleneck regardless of how many total partitions or consumers exist. It's worth reaching for only when the default hash-based distribution genuinely can't express the routing requirement, since uneven partition load is a common and hard-to-diagnose consequence of a poorly-designed custom partitioner.
More Related questions...
