How does Kafka achieve exactly-once semantics with idempotent producers and transactions?

Two separate mechanisms combine to deliver exactly-once processing within Kafka's own read-process-write cycle.

Idempotent producer (enable.idempotence=true) assigns each producer a unique ID and a per-partition sequence number to every batch it sends; if a retry occurs (e.g. due to a network timeout after the broker actually received and wrote the batch), the broker recognizes the duplicate sequence number and discards the retry instead of writing the record twice. This alone solves duplicate writes caused by producer-side retries, but doesn't cover multi-partition atomicity or the consume-process-produce pattern.

enable.idempotence=true
transactional.id=order-processor-1

Transactions build on top of idempotence to make a set of writes — potentially spanning multiple partitions or topics, and including a consumer's offset commit — atomic: either all of them become visible to consumers reading with isolation.level=read_committed, or none do. This is what lets a "read from topic A, process, write to topic B, commit offset" application behave as a single atomic unit, so a crash mid-cycle can't leave the offset committed without the corresponding write, or vice versa — the classic source of duplicate or lost processing that exactly-once semantics are meant to eliminate.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.