AI / LLM Basics Interview Questions
What is Multi-Head Attention?
Multi-Head Attention runs several self-attention operations in parallel, each with its own learned parameters, then combines their results together.
- Each "head" can learn to focus on a different kind of relationship, one might track grammatical structure, another might track topical relevance
- Running these in parallel, rather than one attention operation alone, lets the model capture multiple types of relationships between tokens at once
- The outputs from all heads are concatenated and passed through a final linear layer to produce one combined representation
This is a standard part of every Transformer block, and it's a major reason a single attention layer can capture such a rich, multi-faceted understanding of a sequence.
What does each attention head in Multi-Head Attention potentially focus on?
What happens to the outputs of all the heads?
More Related questions...
