Web / Apache Lucene Interview questions
1. What is Apache Lucene?
Apache Lucene is a free, open-source Java library for full-text indexing and search - not a standalone server. It gives developers the building blocks (indexing, tokenizing, scoring, retrieval) needed to add search to an application, but you embed it directly...
2. What is an inverted index in Lucene?
An inverted index flips the natural document-to-word relationship: instead of storing "document 1 contains these words," it stores "this word appears in these documents." That's what makes searching millions of documents for a term nearly instant instead of scanning each...
3. What is a Lucene Document?
A Document is Lucene's basic unit of indexing and retrieval - roughly analogous to a row in a database table, except it's schema-flexible. Each Document is simply a container holding one or more named Field objects....
4. What are Fields in a Lucene Document?
A Field is a named piece of data attached to a Document - for example title , body , or price . Each Field carries both a value and a FieldType that controls exactly how Lucene treats it....
5. What is an Analyzer in Lucene?
An Analyzer is the pipeline that turns raw text into the stream of terms Lucene actually indexes. It's applied both when documents are indexed and, typically, when queries are parsed - so search terms line up with indexed terms....
6. What is a Tokenizer in Lucene?
A Tokenizer is the first stage of an Analyzer - it reads a raw stream of characters and breaks it into individual tokens (usually words). It does not modify token content; that's left to TokenFilters further down the chain....
7. What is a TokenFilter in Lucene?
A TokenFilter takes the token stream produced by a Tokenizer (or a previous filter) and transforms it further. Filters are chained one after another, and order matters - each one sees only the output of the one before it....
8. What is the purpose of the IndexWriter class?
IndexWriter is the class that creates and modifies a Lucene index - it's the only way to add, update, or delete Documents. Under the hood it buffers incoming Documents in memory and periodically flushes them to disk as new segments....
9. What is the purpose of the IndexSearcher class?
IndexSearcher is the entry point for running queries against an index. It wraps an IndexReader , which provides a consistent, read-only snapshot of the index at a point in time....
10. What are the different types of Field in Lucene?
Lucene ships several ready-made Field subclasses so you don't have to hand-configure a FieldType for common cases: Field Type Behavior TextField Analyzed and tokenized, optionally stored - for prose. StringField Indexed as a single un-tokenized term - for exact-match values...
11. What is a Lucene Directory?
Directory is Lucene's abstraction over where index files physically live. Both IndexWriter and IndexReader talk to a Directory rather than to the filesystem directly, which is what lets the same indexing code run against different storage backends unchanged....
12. Define a Lucene Segment?
A segment is a self-contained, immutable mini-index. A full Lucene index is just a collection of segments plus a small commit file listing which ones are current....
13. Describe the role of the QueryParser in Lucene?
QueryParser converts a human-typed query string, like title:lucene AND status:published , into a Lucene Query object tree that IndexSearcher can actually execute. It understands a compact syntax supporting field prefixes, boolean operators ( AND , OR , NOT ), phrase...
14. List common built-in Analyzers in Lucene?
Lucene ships several ready-made Analyzers so common tokenization needs don't require assembling a custom chain: Analyzer Behavior StandardAnalyzer Unicode word splitting, lowercasing, English stopword removal. WhitespaceAnalyzer Splits only on whitespace, no lowercasing or stopwords.
15. What is a Term in Lucene?
A Term is the atomic unit the inverted index is built from - it's a pair of a field name and a text value, for example (content, "search") . The same word appearing in two different fields is two different...
16. What is the difference between StringField and TextField?
These two Field types look similar but serve opposite purposes, and mixing them up is a frequent source of confusing search behavior. StringField TextField Indexed as a single, un-tokenized term....
17. What is the difference between IndexWriter and IndexWriterConfig?
These two classes split responsibility between action and configuration. IndexWriter is the object that actually performs the work - adding, updating, and deleting documents, and coordinating merges and commits.
18. Why do we use Analyzers with different tokenization strategies?
Different fields carry fundamentally different kinds of data, and one tokenization strategy can't serve all of them well. Prose needs to be split into meaningful words, but an identifier like a SKU or email address needs to stay intact to...
19. How does Lucene score documents (TF-IDF vs BM25)?
Lucene ranks matching documents using a Similarity implementation, and since Lucene 6 the default has been BM25Similarity , replacing the older classic TF-IDF-based vector space model. Both approaches reward documents where a query term appears frequently (term frequency) and penalize...
20. When should you use StandardAnalyzer vs a custom Analyzer?
StandardAnalyzer is a solid default for general-purpose English (and reasonably good multilingual) text: Unicode-aware tokenization, lowercasing, and English stopword removal cover the majority of "search this article/product description" use cases out of the box. A custom Analyzer earns its comp
21. What is the difference between a TermQuery and a PhraseQuery?
Both operate on exact Terms rather than doing fuzzy or analyzed matching at query time, but they check for very different conditions. TermQuery PhraseQuery Matches documents containing a single exact Term....
22. How does the inverted index handle updates and deletes?
Because Lucene segments are immutable once written, there's no in-place editing of a document's terms. Instead, both updates and deletes work around that immutability: Delete - the document is flagged in the segment's live-docs bitset; its postings entries stay physically...
23. Explain the lifecycle of an IndexWriter commit?
A commit is what makes indexing changes durable and visible to newly opened readers. Understanding the stages helps explain why a commit is more expensive than a plain flush.
24. What happens when you call IndexWriter.forceMerge()?
forceMerge() tells Lucene to merge segments down to a target maximum count (often 1) regardless of what the normal MergePolicy would decide on its own. It's the manual override for "consolidate everything now."...
25. How do you optimize a Lucene index for search performance?
Search performance tuning in Lucene usually comes down to a handful of high-leverage decisions rather than one silver bullet: Choose field types deliberately - use DocValues fields for sorting/faceting instead of loading stored values at query time. Disable norms/term vectors...
26. What is the difference between Stored fields and Indexed fields?
"Stored" and "indexed" are two independent switches on a Field, and confusing them is one of the more common Lucene mistakes. Indexed Stored Value is analyzed and added to the inverted index....
27. Why should you use Norms and when can they be disabled?
Norms are small per-document, per-field values that factor field length and any index-time boost into scoring - they're part of why a term match in a short title typically scores higher than the same term buried in a long body...
28. How does Lucene handle segment merging?
Merging is how Lucene keeps segment count under control and reclaims space from deleted documents, since individual segments are never edited in place. A MergePolicy decides which segments to combine and when , while a MergeScheduler decides how that merge...
29. What is the difference between NRT search and a normal commit?
Near Real-Time (NRT) search lets you open an IndexReader directly from an IndexWriter's in-memory state, seeing recently added documents in milliseconds - without going through a full, durable commit first. A normal commit() fsyncs all segment files and writes a...
30. How do you troubleshoot slow queries in Lucene?
Slow-query investigation in Lucene usually follows a fairly consistent checklist rather than guesswork: Inspect the query shape - leading wildcards ( *term ) and broad fuzzy queries are inherently expensive since they can't use the term dictionary efficiently. Check field...
31. Explain the difference between BooleanQuery and BooleanClause?
A BooleanQuery is a composite query that combines other queries logically - it's how Lucene expresses "this AND that, but NOT this other thing." It doesn't do any matching on its own; it's a container....
32. What is the difference between Lucene and Elasticsearch or Solr?
Lucene, Elasticsearch, and Solr are often mentioned together, but they sit at different layers of the stack. Lucene Elasticsearch / Solr A Java library, embedded directly in your process....
33. How does faceting work conceptually in Lucene?
Faceting answers "how many results fall into each category" alongside a normal search - like showing "Electronics (42), Books (17)" next to a product search. Lucene's facet module supports this without scanning every matching document at query time....
34. Which is better and why: FuzzyQuery vs WildcardQuery for typo tolerance?
Neither is universally "better" - they solve different problems, and picking the wrong one for typo tolerance produces frustrating results. FuzzyQuery matches terms within a bounded edit distance (insertions, deletions, substitutions), which is exactly what typos are - it correctly...
35. Explain the execution flow of a search request in Lucene?
From the moment a query object is submitted to the moment ranked results come back, Lucene runs through a consistent per-segment pipeline: flowchart LR A[Query object] --> B[IndexSearcher.search] B --> C[Weight created for query] C --> D[Per-segment Scorer] D -->...
36. Explain the internal working of Lucene's BM25Similarity?
BM25Similarity scores a term match for a document using three interacting components combined multiplicatively, rather than the simpler additive TF-IDF formula it replaced as Lucene's default. IDF (inverse document frequency) - terms appearing in fewer documents across the corpus get...
37. Explain the internal working of segment merging and merge policies?
Merging physically rewrites several existing segments into one new, larger segment: postings lists are combined, deleted documents are dropped entirely, and shared structures like the term dictionary are rebuilt for the merged set. The old segments are only deleted once...
38. How do you implement a custom Analyzer chain?
Building a custom Analyzer means subclassing Analyzer and overriding createComponents() to wire together a Tokenizer with whatever TokenFilters the use case needs, returning them as TokenStreamComponents . public class ProductCodeAnalyzer extends Analyzer { & # 64 ;Override protected TokenStreamC
39. Explain the internal working of Lucene's codec architecture?
A Codec defines exactly how every part of a segment is physically encoded on disk - postings lists, stored fields, term vectors, doc values, norms, and point (BKD tree) data each have their own file format, and the Codec is...
40. How can you optimize indexing throughput for large-scale data?
Indexing throughput tuning trades some memory, durability latency, or feature richness for raw speed, so each lever below should be chosen deliberately rather than applied blindly: Increase the RAM buffer size ( IndexWriterConfig.setRAMBufferSizeMB ) so more documents accumulate before a...
41. Explain the lifecycle of an IndexSearcher across an NRT reopen?
Rather than closing and reopening searchers manually, Lucene's SearcherManager (a ReferenceManager implementation) manages this lifecycle so no in-flight search ever sees a reader closed out from under it. sequenceDiagram participant App participant SearcherManager participant OldSearcher partici
42. What is the difference between DocValues and stored fields for sorting or faceting?
Both can technically hold the same data, but they're organized completely differently on disk, and that difference matters a lot for sorting and faceting workloads specifically. Stored fields DocValues Row-oriented: all fields for one document stored together....
43. Explain the internal working of Lucene's point-based fields (BKD tree) for range queries?
Since Lucene 6, numeric and spatial range queries (IntPoint, LongPoint, DoublePoint, and spatial fields) are backed by a BKD tree (a block K-dimensional tree), which replaced the older trie-based NumericField encoding used previously. A BKD tree recursively partitions points into...
44. How do you implement custom scoring using Lucene's Similarity API?
When BM25's general-purpose relevance model doesn't capture business-specific ranking needs - like factoring in a product's popularity alongside text relevance - Lucene lets you plug in a custom Similarity implementation, typically by extending SimilarityBase or wrapping the default with a...
45. Explain the internal working of the SpanQuery family?
Most Lucene queries (TermQuery, BooleanQuery) care only about whether and how often terms match - not precisely where relative to each other. SpanQueries are the exception: they operate directly on term positions , matching contiguous or constrained ranges within a...
46. How does Lucene ensure durability and crash recovery?
Lucene's durability model is built entirely around the commit point concept rather than a separate write-ahead log layer - that additional layer (like a translog) is something systems built on Lucene, such as Elasticsearch, add themselves. Each commit writes a...
47. What is the difference between per-field similarity and global similarity configuration?
By default, a single Similarity instance (BM25Similarity, unless changed) is set once on IndexWriterConfig and used uniformly across every field in the index - that's global configuration. Per-field similarity , via PerFieldSimilarityWrapper , lets different fields use entirely different scoring
48. Explain the internal working of Lucene's MMapDirectory I/O?
MMapDirectory reads index files using the operating system's memory-mapping facility ( mmap ) instead of issuing explicit read() system calls the way NIOFSDirectory does. The file's contents are mapped directly into the process's virtual address space, and pages are faulted...
49. How do you troubleshoot OOM errors in a Lucene-based application?
Out-of-memory errors in a Lucene application usually trace back to one of a small set of causes, so a systematic pass through them beats guessing: Check the RAM buffer and indexing thread count - a large RAM buffer combined with...
50. Explain how Lucene's architecture influences distributed search systems like Solr and Elasticsearch?
Lucene's segment-based, immutable-write model turns out to map remarkably cleanly onto distributed systems concepts, which is a big part of why both Solr and Elasticsearch were built directly on top of it rather than writing their own indexing engine. Because...
