AI / LLM Basics Interview Questions
What is the purpose of the Feed-Forward Network inside a Transformer block?
Each Transformer block contains a feed-forward network that processes every token's representation individually after the attention step has mixed in context from the rest of the sequence.
- Applies the same two-layer transformation to each token's vector independently
- Adds additional representational capacity and non-linearity that attention alone doesn't provide
- Works alongside attention in every block, attention gathers context across tokens, the feed-forward network then further processes each token's resulting representation
Together, the attention and feed-forward steps in each block are what let a Transformer refine its understanding of a sequence layer by layer as information flows through the network.
When does the feed-forward network process a token's representation?
Does the feed-forward network process tokens individually or all mixed together?
More Related questions...
