AI / LLM Basics Interview Questions
What is Byte-Pair Encoding (BPE)?
Byte-Pair Encoding is a popular tokenization algorithm that builds a vocabulary by repeatedly merging the most frequently occurring pair of characters or character sequences in a training corpus.
- Starts with individual characters as the smallest units
- Iteratively merges the most common adjacent pairs into new, longer tokens
- Stops once a target vocabulary size is reached, typically tens of thousands of tokens
This approach strikes a practical balance: common words end up as single tokens for efficiency, while rare or unseen words can still be represented by breaking them into smaller, previously-learned pieces instead of failing entirely.
What does BPE build its vocabulary by doing?
What happens to rare or unseen words under BPE?
More Related questions...
