Why doesn't increasing partition count always improve throughput?

Partitions raise the theoretical ceiling on parallelism, but actual throughput depends on more than just that ceiling — several other factors can mean adding partitions delivers little or no real improvement, or even makes things worse.

• Consumer count already below partition count — if a consumer group isn't running enough instances to use the partitions it already has, adding more partitions doesn't help until consumer count also increases.
• Downstream bottleneck, not Kafka — if each consumer's processing is limited by a slow database write or external API call, more partitions just create more parallel callers hitting that same downstream bottleneck.
• Per-broker overhead — every partition adds file handles, replication traffic, and controller metadata load; beyond a certain point, more partitions can degrade controller and broker performance rather than improving client-facing throughput.
• Producer batching efficiency — spreading the same total write volume across many more partitions means smaller batches per partition, which can reduce the batching efficiency that drives producer throughput in the first place.

The practical takeaway: partition count is one input to throughput, not the whole story, and increasing it without also addressing consumer capacity or a genuine downstream bottleneck is a common, ineffective first response to a throughput problem.