What is Multi-Head Attention?

Multi-Head Attention runs several self-attention operations in parallel, each with its own learned parameters, then combines their results together.

• Each "head" can learn to focus on a different kind of relationship, one might track grammatical structure, another might track topical relevance
• Running these in parallel, rather than one attention operation alone, lets the model capture multiple types of relationships between tokens at once
• The outputs from all heads are concatenated and passed through a final linear layer to produce one combined representation

This is a standard part of every Transformer block, and it's a major reason a single attention layer can capture such a rich, multi-faceted understanding of a sequence.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.