Web / Apache Lucene Interview questions
How do you implement a custom Analyzer chain?
Building a custom Analyzer means subclassing Analyzer and overriding createComponents() to wire together a Tokenizer with whatever TokenFilters the use case needs, returning them as TokenStreamComponents.
public class ProductCodeAnalyzer extends Analyzer { @Override protected TokenStreamComponents createComponents(String fieldName) { Tokenizer source = new WhitespaceTokenizer(); TokenStream filtered = new LowerCaseFilter(source); filtered = new StopFilter(filtered, EnglishAnalyzer.ENGLISH_STOP_WORDS_SET); filtered = new PorterStemFilter(filtered); return new TokenStreamComponents(source, filtered); } }
Each filter wraps the previous TokenStream, so order matters - lowercasing before stopword removal ensures "The" is recognized as a stopword just like "the". The returned source is what actually receives characters, while filtered (the last filter in the chain) is what IndexWriter and QueryParser actually consume.
Custom Analyzers are typically registered per field via PerFieldAnalyzerWrapper, so different fields in the same Document can each use the Analyzer that fits their content.
Which method must be overridden to define a custom Analyzer's chain?
Applying different Analyzers to different fields in the same Document is done via:
More Related questions...
