Integration / Apache Kafka Interview questions
When should you increase the number of partitions for a topic?
More partitions raise the ceiling on parallel throughput — both how many producers can write concurrently without contention and how many consumers in a group can process in parallel, since consumer parallelism within a group is capped at the partition count. Increasing partitions makes sense when a topic's current consumer group can't keep up because it's already running as many consumer instances as there are partitions, or when a single partition's write throughput is approaching what one broker can realistically sustain.
It's not a free lever, though, and shouldn't be increased casually:
- More partitions means more open file handles and more replication traffic per broker, adding overhead cluster-wide.
- Partition count can only go up, never down, on an existing topic — a topic over-provisioned with partitions can't be shrunk back later without recreating it.
- Increasing partitions on an existing topic changes the key-to-partition mapping going forward, which can break ordering guarantees for keys that previously mapped consistently to one partition.
The practical guidance: size partition count for expected peak parallelism up front where possible, and treat later increases as a deliberate, carefully-considered change rather than a routine scaling knob.
Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!
Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.
Invest now!!! Get Free equity stock (US, UK only)!
Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.
The Robinhood app makes it easy to trade stocks, crypto and more.
Webull! Receive free stock by signing up using the link: Webull signup.
More Related questions...
