What is a Kafka offset used for?

An offset is a sequential, per-partition number Kafka assigns to each record as it's appended to the log — 0, 1, 2, and so on, within that specific partition. Its main job is tracking consumption progress: a consumer records which offset it has processed up to in each assigned partition, so it knows exactly where to resume if it restarts or a partition gets reassigned to a different consumer during a rebalance.

consumer.commitSync();  // commits the offsets of the most recently polled records

Consumers can commit offsets automatically on an interval (enable.auto.commit=true) or manually via commitSync()/commitAsync() for tighter control over exactly when "processed" is recorded relative to actual work being done — a distinction that directly determines whether an application behaves closer to at-least-once or at-most-once delivery. Committed offsets themselves are stored durably in an internal Kafka topic (__consumer_offsets), not on the consumer's local machine, so progress survives a consumer restarting on a different host entirely.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.