Web / Apache Lucene Interview questions
1. What is Apache Lucene?
Apache Lucene is a free, open-source Java library for full-text indexing and search - not a standalone server. It gives developers the building blocks (indexing, tokenizing, scoring, retrieval) needed to add search to an application, but you embed it directly in code rather than talking to it ove...
2. What is an inverted index in Lucene?
An inverted index flips the natural document-to-word relationship: instead of storing "document 1 contains these words," it stores "this word appears in these documents." That's what makes searching millions of documents for a term nearly instant instead of scanning each one. For every unique ter...
3. What is a Lucene Document?
A Document is Lucene's basic unit of indexing and retrieval - roughly analogous to a row in a database table, except it's schema-flexible. Each Document is simply a container holding one or more named Field objects. You don't index raw files or objects directly; you construct a Document, add Fiel...
4. What are Fields in a Lucene Document?
A Field is a named piece of data attached to a Document - for example title , body , or price . Each Field carries both a value and a FieldType that controls exactly how Lucene treats it. The FieldType settings decide three independent things: Indexed - whether the value is analyzed and added to ...
5. What is an Analyzer in Lucene?
An Analyzer is the pipeline that turns raw text into the stream of terms Lucene actually indexes. It's applied both when documents are indexed and, typically, when queries are parsed - so search terms line up with indexed terms. Internally, an Analyzer chains together exactly one Tokenizer (split...
6. What is a Tokenizer in Lucene?
A Tokenizer is the first stage of an Analyzer - it reads a raw stream of characters and breaks it into individual tokens (usually words). It does not modify token content; that's left to TokenFilters further down the chain. Common built-in tokenizers include: StandardTokenizer - Unicode-aware wor...
7. What is a TokenFilter in Lucene?
A TokenFilter takes the token stream produced by a Tokenizer (or a previous filter) and transforms it further. Filters are chained one after another, and order matters - each one sees only the output of the one before it. Common examples include: LowerCaseFilter - normalizes case so "Lucene" and ...
8. What is the purpose of the IndexWriter class?
IndexWriter is the class that creates and modifies a Lucene index - it's the only way to add, update, or delete Documents. Under the hood it buffers incoming Documents in memory and periodically flushes them to disk as new segments. Key responsibilities include: Analyzing and adding new Documents...
9. What is the purpose of the IndexSearcher class?
IndexSearcher is the entry point for running queries against an index. It wraps an IndexReader , which provides a consistent, read-only snapshot of the index at a point in time. Given a Query object, IndexSearcher builds a Weight and Scorer per segment, collects matching documents, ranks them usi...
10. What are the different types of Field in Lucene?
Lucene ships several ready-made Field subclasses so you don't have to hand-configure a FieldType for common cases: Field Type Behavior TextField Analyzed and tokenized, optionally stored - for prose. StringField Indexed as a single un-tokenized term - for exact-match values like IDs. StoredField ...
11. What is a Lucene Directory?
Directory is Lucene's abstraction over where index files physically live. Both IndexWriter and IndexReader talk to a Directory rather than to the filesystem directly, which is what lets the same indexing code run against different storage backends unchanged. The most common implementations are: F...
12. Define a Lucene Segment?
A segment is a self-contained, immutable mini-index. A full Lucene index is just a collection of segments plus a small commit file listing which ones are current. Every time IndexWriter flushes buffered documents, it writes a brand-new segment rather than editing an existing one - segments, once ...
13. Describe the role of the QueryParser in Lucene?
QueryParser converts a human-typed query string, like title:lucene AND status:published , into a Lucene Query object tree that IndexSearcher can actually execute. It understands a compact syntax supporting field prefixes, boolean operators ( AND , OR , NOT ), phrase queries in quotes, wildcards, ...
14. List common built-in Analyzers in Lucene?
Lucene ships several ready-made Analyzers so common tokenization needs don't require assembling a custom chain: Analyzer Behavior StandardAnalyzer Unicode word splitting, lowercasing, English stopword removal. WhitespaceAnalyzer Splits only on whitespace, no lowercasing or stopwords. SimpleAnalyz...
15. What is a Term in Lucene?
A Term is the atomic unit the inverted index is built from - it's a pair of a field name and a text value, for example (content, "search") . The same word appearing in two different fields is two different Terms. Every entry in the inverted index is keyed by a Term, and each Term points to a post...
16. What is the difference between StringField and TextField?
These two Field types look similar but serve opposite purposes, and mixing them up is a frequent source of confusing search behavior. StringField TextField Indexed as a single, un-tokenized term. Passed through the Analyzer and tokenized into many terms. Good for exact-match values: IDs, status c...
17. What is the difference between IndexWriter and IndexWriterConfig?
These two classes split responsibility between action and configuration. IndexWriter is the object that actually performs the work - adding, updating, and deleting documents, and coordinating merges and commits. IndexWriterConfig is a settings object handed to the IndexWriter's constructor, contr...
18. Why do we use Analyzers with different tokenization strategies?
Different fields carry fundamentally different kinds of data, and one tokenization strategy can't serve all of them well. Prose needs to be split into meaningful words, but an identifier like a SKU or email address needs to stay intact to remain searchable as a whole. Using StandardAnalyzer on a ...
19. How does Lucene score documents (TF-IDF vs BM25)?
Lucene ranks matching documents using a Similarity implementation, and since Lucene 6 the default has been BM25Similarity , replacing the older classic TF-IDF-based vector space model. Both approaches reward documents where a query term appears frequently (term frequency) and penalize terms that ...
20. When should you use StandardAnalyzer vs a custom Analyzer?
StandardAnalyzer is a solid default for general-purpose English (and reasonably good multilingual) text: Unicode-aware tokenization, lowercasing, and English stopword removal cover the majority of "search this article/product description" use cases out of the box. A custom Analyzer earns its comp...
21. What is the difference between a TermQuery and a PhraseQuery?
Both operate on exact Terms rather than doing fuzzy or analyzed matching at query time, but they check for very different conditions. TermQuery PhraseQuery Matches documents containing a single exact Term. Matches documents containing a sequence of Terms in order. Ignores term position entirely. ...
22. How does the inverted index handle updates and deletes?
Because Lucene segments are immutable once written, there's no in-place editing of a document's terms. Instead, both updates and deletes work around that immutability: Delete - the document is flagged in the segment's live-docs bitset; its postings entries stay physically on disk but are skipped ...
23. Explain the lifecycle of an IndexWriter commit?
A commit is what makes indexing changes durable and visible to newly opened readers. Understanding the stages helps explain why a commit is more expensive than a plain flush. sequenceDiagram participant App participant IndexWriter participant Directory App->>IndexWriter: addDocument()/updateDocum...
24. What happens when you call IndexWriter.forceMerge()?
forceMerge() tells Lucene to merge segments down to a target maximum count (often 1) regardless of what the normal MergePolicy would decide on its own. It's the manual override for "consolidate everything now." Two concrete effects follow from that: first, deleted documents in the merged-away seg...
25. How do you optimize a Lucene index for search performance?
Search performance tuning in Lucene usually comes down to a handful of high-leverage decisions rather than one silver bullet: Choose field types deliberately - use DocValues fields for sorting/faceting instead of loading stored values at query time. Disable norms/term vectors on fields where scor...
26. What is the difference between Stored fields and Indexed fields?
"Stored" and "indexed" are two independent switches on a Field, and confusing them is one of the more common Lucene mistakes. Indexed Stored Value is analyzed and added to the inverted index. Original value is kept in a separate store for retrieval. Makes the field searchable . Makes the field re...
27. Why should you use Norms and when can they be disabled?
Norms are small per-document, per-field values that factor field length and any index-time boost into scoring - they're part of why a term match in a short title typically scores higher than the same term buried in a long body field. Storing norms costs a small amount of memory per field per docu...
28. How does Lucene handle segment merging?
Merging is how Lucene keeps segment count under control and reclaims space from deleted documents, since individual segments are never edited in place. A MergePolicy decides which segments to combine and when , while a MergeScheduler decides how that merge work is actually executed (usually on ba...
29. What is the difference between NRT search and a normal commit?
Near Real-Time (NRT) search lets you open an IndexReader directly from an IndexWriter's in-memory state, seeing recently added documents in milliseconds - without going through a full, durable commit first. A normal commit() fsyncs all segment files and writes a new segments_N commit point, which...
30. How do you troubleshoot slow queries in Lucene?
Slow-query investigation in Lucene usually follows a fairly consistent checklist rather than guesswork: Inspect the query shape - leading wildcards ( *term ) and broad fuzzy queries are inherently expensive since they can't use the term dictionary efficiently. Check field analysis - a mismatched ...
31. Explain the difference between BooleanQuery and BooleanClause?
A BooleanQuery is a composite query that combines other queries logically - it's how Lucene expresses "this AND that, but NOT this other thing." It doesn't do any matching on its own; it's a container. Each BooleanClause is one entry inside that container: a sub-query paired with an Occur value t...
32. What is the difference between Lucene and Elasticsearch or Solr?
Lucene, Elasticsearch, and Solr are often mentioned together, but they sit at different layers of the stack. Lucene Elasticsearch / Solr A Java library, embedded directly in your process. Standalone server platforms, accessed over HTTP/REST. No built-in networking, clustering, or sharding. Provid...
33. How does faceting work conceptually in Lucene?
Faceting answers "how many results fall into each category" alongside a normal search - like showing "Electronics (42), Books (17)" next to a product search. Lucene's facet module supports this without scanning every matching document at query time. The typical approach uses a SortedSetDocValuesF...
34. Which is better and why: FuzzyQuery vs WildcardQuery for typo tolerance?
Neither is universally "better" - they solve different problems, and picking the wrong one for typo tolerance produces frustrating results. FuzzyQuery matches terms within a bounded edit distance (insertions, deletions, substitutions), which is exactly what typos are - it correctly matches "lucne...
35. Explain the execution flow of a search request in Lucene?
From the moment a query object is submitted to the moment ranked results come back, Lucene runs through a consistent per-segment pipeline: flowchart LR A[Query object] --> B[IndexSearcher.search] B --> C[Weight created for query] C --> D[Per-segment Scorer] D --> E[Collector gathers matches] E --...
36. Explain the internal working of Lucene's BM25Similarity?
BM25Similarity scores a term match for a document using three interacting components combined multiplicatively, rather than the simpler additive TF-IDF formula it replaced as Lucene's default. IDF (inverse document frequency) - terms appearing in fewer documents across the corpus get a higher wei...
37. Explain the internal working of segment merging and merge policies?
Merging physically rewrites several existing segments into one new, larger segment: postings lists are combined, deleted documents are dropped entirely, and shared structures like the term dictionary are rebuilt for the merged set. The old segments are only deleted once the new one is fully writt...
38. How do you implement a custom Analyzer chain?
Building a custom Analyzer means subclassing Analyzer and overriding createComponents() to wire together a Tokenizer with whatever TokenFilters the use case needs, returning them as TokenStreamComponents . public class ProductCodeAnalyzer extends Analyzer { & # 64 ;Override protected TokenStreamC...
39. Explain the internal working of Lucene's codec architecture?
A Codec defines exactly how every part of a segment is physically encoded on disk - postings lists, stored fields, term vectors, doc values, norms, and point (BKD tree) data each have their own file format, and the Codec is the pluggable component that determines those formats. Lucene ships a def...
40. How can you optimize indexing throughput for large-scale data?
Indexing throughput tuning trades some memory, durability latency, or feature richness for raw speed, so each lever below should be chosen deliberately rather than applied blindly: Increase the RAM buffer size ( IndexWriterConfig.setRAMBufferSizeMB ) so more documents accumulate before a flush, r...
41. Explain the lifecycle of an IndexSearcher across an NRT reopen?
Rather than closing and reopening searchers manually, Lucene's SearcherManager (a ReferenceManager implementation) manages this lifecycle so no in-flight search ever sees a reader closed out from under it. sequenceDiagram participant App participant SearcherManager participant OldSearcher partici...
42. What is the difference between DocValues and stored fields for sorting or faceting?
Both can technically hold the same data, but they're organized completely differently on disk, and that difference matters a lot for sorting and faceting workloads specifically. Stored fields DocValues Row-oriented: all fields for one document stored together. Column-oriented: all values for one ...
43. Explain the internal working of Lucene's point-based fields (BKD tree) for range queries?
Since Lucene 6, numeric and spatial range queries (IntPoint, LongPoint, DoublePoint, and spatial fields) are backed by a BKD tree (a block K-dimensional tree), which replaced the older trie-based NumericField encoding used previously. A BKD tree recursively partitions points into a balanced tree ...
44. How do you implement custom scoring using Lucene's Similarity API?
When BM25's general-purpose relevance model doesn't capture business-specific ranking needs - like factoring in a product's popularity alongside text relevance - Lucene lets you plug in a custom Similarity implementation, typically by extending SimilarityBase or wrapping the default with a PerFie...
45. Explain the internal working of the SpanQuery family?
Most Lucene queries (TermQuery, BooleanQuery) care only about whether and how often terms match - not precisely where relative to each other. SpanQueries are the exception: they operate directly on term positions , matching contiguous or constrained ranges within a document. Key building blocks i...
46. How does Lucene ensure durability and crash recovery?
Lucene's durability model is built entirely around the commit point concept rather than a separate write-ahead log layer - that additional layer (like a translog) is something systems built on Lucene, such as Elasticsearch, add themselves. Each commit writes a new segments_N file only after all r...
47. What is the difference between per-field similarity and global similarity configuration?
By default, a single Similarity instance (BM25Similarity, unless changed) is set once on IndexWriterConfig and used uniformly across every field in the index - that's global configuration. Per-field similarity , via PerFieldSimilarityWrapper , lets different fields use entirely different scoring ...
48. Explain the internal working of Lucene's MMapDirectory I/O?
MMapDirectory reads index files using the operating system's memory-mapping facility ( mmap ) instead of issuing explicit read() system calls the way NIOFSDirectory does. The file's contents are mapped directly into the process's virtual address space, and pages are faulted in from disk by the OS...
49. How do you troubleshoot OOM errors in a Lucene-based application?
Out-of-memory errors in a Lucene application usually trace back to one of a small set of causes, so a systematic pass through them beats guessing: Check the RAM buffer and indexing thread count - a large RAM buffer combined with many concurrent indexing threads multiplies memory pressure at write...
50. Explain how Lucene's architecture influences distributed search systems like Solr and Elasticsearch?
Lucene's segment-based, immutable-write model turns out to map remarkably cleanly onto distributed systems concepts, which is a big part of why both Solr and Elasticsearch were built directly on top of it rather than writing their own indexing engine. Because each Lucene index is already self-con...
