Web / Apache Lucene Interview questions
How does Lucene ensure durability and crash recovery?
Lucene's durability model is built entirely around the commit point concept rather than a separate write-ahead log layer - that additional layer (like a translog) is something systems built on Lucene, such as Elasticsearch, add themselves.
- Each commit writes a new
segments_Nfile only after all referenced segment files have been fsynced to disk, guaranteeing the data those segments reference is actually durable first. - The
segments_Nfile write itself is atomic at the filesystem level (write-then-rename), so a reader never observes a half-written commit point. - An IndexDeletionPolicy decides how many past commit points to retain; by default only the most recent is kept, but retaining older ones enables point-in-time recovery or external backup strategies.
- On startup, Lucene simply opens the most recent valid
segments_N- any segment files written after the last successful commit but before a crash are just orphaned files, never referenced, and can be safely ignored or cleaned up.
The trade-off is that anything indexed after the last commit and not yet durable is lost on crash - which is exactly why systems needing stronger guarantees between commits, like Elasticsearch, layer their own translog on top rather than relying on Lucene's commit granularity alone.
More Related questions...