Explain the internal working of ActiveMQ's KahaDB persistence store?

KahaDB is ActiveMQ's default file-based persistence adapter. It combines a write-ahead log with an index so persistent messages survive broker restarts without requiring random-access disk writes for every operation.

When a persistent message arrives, KahaDB appends it to the current active data log file sequentially, which is cheap because it requires no disk seek, then updates its page-file index entry pointing to that offset. Message removal after acknowledgment doesn't rewrite the log entry in place; instead it marks the entry as deleted in the index. A background process periodically checks whether every message in an older data log file has been marked deleted, and once a whole file is fully consumed, KahaDB deletes that entire file rather than compacting individual records, keeping the reclaim cost low.

On broker restart, KahaDB replays the data logs to rebuild its index, which is why very large uncompacted logs can slow down startup after an unclean shutdown. This log-structured design trades some read-path indirection for very fast sequential writes, which is why KahaDB generally outperforms JDBC persistence for a single, locally-attached broker, though it doesn't natively support shared storage across multiple broker instances the way a database can.

Because reclaiming a file requires every message in it to be fully consumed first, a single long-lived unacknowledged message can indefinitely pin down an entire data log file, which is a common, if underappreciated, cause of disk usage that keeps climbing even though most traffic is flowing normally.