Prev Next

Web / Apache Lucene Interview questions

Explain how Lucene's architecture influences distributed search systems like Solr and Elasticsearch?

Lucene's segment-based, immutable-write model turns out to map remarkably cleanly onto distributed systems concepts, which is a big part of why both Solr and Elasticsearch were built directly on top of it rather than writing their own indexing engine.

Because each Lucene index is already self-contained and independent, it's a natural unit of horizontal scaling: Elasticsearch's shards and Solr's cores/shards are each just one Lucene index, so scaling out means running more independent Lucene instances rather than redesigning the indexing engine itself. Replication follows the same logic - a replica shard is simply another Lucene index kept in sync with its primary, often by shipping the same indexing operations to both.

The immutable-segment model also fits distributed replication well: since segments are never modified in place, replicating an index (or shipping incremental segment files to a replica) is a matter of copying whole, unchanging files rather than reconciling in-place edits, which simplifies consistency compared to a mutable on-disk structure.

What Elasticsearch and Solr add on top - cluster coordination, request routing across shards, result merging from multiple nodes, and a translog for stronger durability between Lucene commits - is precisely the layer Lucene deliberately leaves out, since it's designed as an embeddable library rather than a distributed system in its own right.

An Elasticsearch shard or Solr core maps internally to:
Lucene's immutable-segment model simplifies distributed replication because:

More Related questions...

What is Apache Lucene? What is an inverted index in Lucene? What is a Lucene Document? What are Fields in a Lucene Document? What is an Analyzer in Lucene? What is a Tokenizer in Lucene? What is a TokenFilter in Lucene? What is the purpose of the IndexWriter class? What is the purpose of the IndexSearcher class? What are the different types of Field in Lucene? What is a Lucene Directory? Define a Lucene Segment? Describe the role of the QueryParser in Lucene? List common built-in Analyzers in Lucene? What is a Term in Lucene? What is the difference between StringField and TextField? What is the difference between IndexWriter and IndexWriterConfig? Why do we use Analyzers with different tokenization strategies? How does Lucene score documents (TF-IDF vs BM25)? When should you use StandardAnalyzer vs a custom Analyzer? What is the difference between a TermQuery and a PhraseQuery? How does the inverted index handle updates and deletes? Explain the lifecycle of an IndexWriter commit? What happens when you call IndexWriter.forceMerge()? How do you optimize a Lucene index for search performance? What is the difference between Stored fields and Indexed fields? Why should you use Norms and when can they be disabled? How does Lucene handle segment merging? What is the difference between NRT search and a normal commit? How do you troubleshoot slow queries in Lucene? Explain the difference between BooleanQuery and BooleanClause? What is the difference between Lucene and Elasticsearch or Solr? How does faceting work conceptually in Lucene? Which is better and why: FuzzyQuery vs WildcardQuery for typo tolerance? Explain the execution flow of a search request in Lucene? Explain the internal working of Lucene's BM25Similarity? Explain the internal working of segment merging and merge policies? How do you implement a custom Analyzer chain? Explain the internal working of Lucene's codec architecture? How can you optimize indexing throughput for large-scale data? Explain the lifecycle of an IndexSearcher across an NRT reopen? What is the difference between DocValues and stored fields for sorting or faceting? Explain the internal working of Lucene's point-based fields (BKD tree) for range queries? How do you implement custom scoring using Lucene's Similarity API? Explain the internal working of the SpanQuery family? How does Lucene ensure durability and crash recovery? What is the difference between per-field similarity and global similarity configuration? Explain the internal working of Lucene's MMapDirectory I/O? How do you troubleshoot OOM errors in a Lucene-based application? Explain how Lucene's architecture influences distributed search systems like Solr and Elasticsearch?
Show more question and Answers...


Comments & Discussions