AI / LLM Basics Interview Questions
What is Layer Normalization?
Layer Normalization is a technique used inside a Transformer to keep the scale of values flowing through the network stable as they pass through many stacked layers.
- Rescales the values within each layer's output to have a consistent mean and variance
- Helps training converge faster and more reliably, since wildly varying value scales can make gradient-based training unstable
- Applied at specific points within each Transformer block, typically before or after the attention and feed-forward steps
Without this kind of stabilization, training a network as deep as a modern LLM, often dozens of stacked layers, would be considerably harder to get working reliably.
What does Layer Normalization keep stable?
What does Layer Normalization help with during training?
More Related questions...
