Web / Apache Solr Interview questions
Describe tokenizer in Solr?
A tokenizer is the first stage of an analyzer. It takes a raw stream of characters and breaks it into a stream of tokens, typically words, based on rules like whitespace or punctuation.
Common built-in tokenizers include:
- StandardTokenizer - splits on whitespace and punctuation using Unicode text segmentation rules; the general-purpose default.
- WhitespaceTokenizer - splits only on whitespace, keeping punctuation attached.
- KeywordTokenizer - treats the entire input as a single token, useful for exact-match fields.
Only one tokenizer can be used per analyzer, and it always runs before any token filters.
Which tokenizer treats the whole input string as one single token?
Where does the tokenizer sit in the analysis chain?
More Related questions...
