Web / Apache Lucene Interview questions
What is a Tokenizer in Lucene?
A Tokenizer is the first stage of an Analyzer - it reads a raw stream of characters and breaks it into individual tokens (usually words). It does not modify token content; that's left to TokenFilters further down the chain.
Common built-in tokenizers include:
- StandardTokenizer - Unicode-aware word-boundary splitting, the general-purpose default.
- WhitespaceTokenizer - splits only on whitespace, keeping punctuation attached.
- KeywordTokenizer - treats the entire input as a single token, useful for IDs or tags.
- LetterTokenizer - splits on any non-letter character.
Picking the right tokenizer matters: using StandardTokenizer on a product SKU field, for example, could split "SKU-1234" into pieces you never wanted searchable separately.
A Tokenizer's job in the Analyzer chain is to:
Which tokenizer treats the whole input as one single token?
More Related questions...
