AI / LLM Basics Interview Questions
What is Self-Attention?
Self-attention is attention applied within a single sequence, each token attends to every other token in that same input, including itself, rather than attending to a separate sequence.
- Every token generates a Query, Key, and Value vector from its embedding
- The Query of one token is compared against the Keys of every token to compute relevance scores
- Those scores weight each token's Value, producing a new representation that blends in relevant context from across the sequence
This is the specific form of attention used throughout an LLM's internal layers, letting the model build up an increasingly context-aware representation of each token as it passes through the network.
In self-attention, what does each token attend to?
What three vectors does each token generate for self-attention?
More Related questions...
