Integration / Apache Kafka Interview questions
What is the difference between log compaction and log deletion cleanup policies?
Both are ways Kafka reclaims disk space from a topic's log over time, but they remove different things and suit different use cases.
| Deletion (cleanup.policy=delete) | Compaction (cleanup.policy=compact) |
| Removes entire old segments once they exceed retention.ms/retention.bytes. | Removes older records that share a key, keeping only the latest value per key. |
| Good for event streams where only recent history matters. | Good for representing current state, e.g. a changelog of the latest value per entity ID. |
| A key's full history disappears once its segment ages out. | The latest record for every key is retained indefinitely (until a tombstone removes it). |
| Simpler, predictable storage growth based on time/size. | Storage still grows with number of distinct keys, but not with update frequency. |
A common real pattern is a topic backing a Kafka Streams state store or serving as a changelog for a cache: compaction guarantees you can always reconstruct the latest state for every key by reading the topic from the start, since old updates to the same key are eventually removed but the newest one for each key persists. A null value acts as a tombstone, marking a key for eventual removal entirely rather than just superseding its prior value.
Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!
Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.
Invest now!!! Get Free equity stock (US, UK only)!
Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.
The Robinhood app makes it easy to trade stocks, crypto and more.
Webull! Receive free stock by signing up using the link: Webull signup.
More Related questions...
