AI / LLM Basics Interview Questions
What is Tokenization?
Tokenization is the process of converting raw text into the sequence of tokens a model can actually process, since a neural network can't work with plain text directly.
- A tokenizer breaks input text apart based on a fixed vocabulary it was built with
- The same tokenizer is used both to prepare input for the model and to decode the model's output tokens back into readable text
- Different models often use different tokenizers, so a "1,000 token" limit can represent slightly different amounts of actual text between models
Tokenization happens before any of the model's actual reasoning, it's the translation step between human-readable text and the numeric representation the model works with internally.
What does a tokenizer convert text into?
Do all models necessarily use the exact same tokenizer?
More Related questions...
