AI / LLM Basics Interview Questions
What is an Encoder-Only model?
An Encoder-only model processes an entire input sequence at once, with every token able to attend to every other token, including ones that come later in the sequence.
- No causal masking, since it's not generating text one token at a time
- Well suited to understanding tasks like classification, sentiment analysis, or extracting information from text, rather than generating new text
- BERT is the best-known example of this architecture family
Encoder-only models excel at deeply understanding a fixed piece of input text, but they aren't naturally built for the open-ended generation task that decoder-only models handle.
Can tokens in an encoder-only model attend to later tokens?
What is the best-known example of an encoder-only model?
More Related questions...
