Web / Apache Lucene Interview questions
Why do we use Analyzers with different tokenization strategies?
Different fields carry fundamentally different kinds of data, and one tokenization strategy can't serve all of them well. Prose needs to be split into meaningful words, but an identifier like a SKU or email address needs to stay intact to remain searchable as a whole.
Using StandardAnalyzer on a product code such as ABC-1234 could split it into abc and 1234, silently breaking exact lookups on the full code. Conversely, using KeywordAnalyzer on an article body would index the entire text as one giant unsearchable blob.
Matching the Analyzer to the field's actual content and query patterns - prose fields get a linguistic analyzer, identifier fields get KeywordAnalyzer or none at all, and multi-language content may need per-language analyzers - is what keeps both indexing size and search relevance under control.
More Related questions...
