What are the differences between Batch Norm, Layer Norm, Group Norm, and Instance Norm?

All normalisation variants compute mean and variance and apply the same transformation (x-μ)/√(σ²+ε) — they differ only in which dimensions the mean and variance are computed over. This seemingly small difference has large practical consequences depending on the architecture and batch size.

import torch
import torch.nn as nn

# BatchNorm1d: normalise over batch for FC layers
# Input: (N, C) or (N, C, L)
bn = nn.BatchNorm1d(num_features=128)

# LayerNorm: normalise over feature dim(s) — no dependency on batch
# Input: (*, normalized_shape)  — last dims are normalised
ln = nn.LayerNorm(normalized_shape=128)  # used in transformers
ln_2d = nn.LayerNorm([128, 8, 8])        # can normalise spatial too

# GroupNorm: split channels into groups, normalise per group per sample
# Input: (N, C, *)  — C must be divisible by num_groups
gn = nn.GroupNorm(num_groups=8, num_channels=128)

# InstanceNorm: each sample, each channel independently
inst = nn.InstanceNorm2d(num_features=128)

# Example: why LayerNorm is used in transformers
d_model = 512
x = torch.randn(4, 20, d_model)   # (batch, seq_len, d_model)
# BatchNorm would normalise over batch and seq_len per feature dim —
# unstable at inference when batch=1 (as in autoregressive generation)
# LayerNorm normalises over d_model for each (batch, seq) position independently
print(ln(x).shape)  # (4, 20, 512) — each position normalised independently