What is weight initialization in PyTorch and why does it matter?

How a network's weights are initialised at the start of training significantly affects whether training converges quickly, slowly, or not at all. PyTorch's default initialisation (Kaiming uniform for Linear/Conv layers) works well in most cases, but understanding the principles helps when debugging training issues.

import torch
import torch.nn as nn

# PyTorch default: Linear layers use Kaiming Uniform initialisation
layer = nn.Linear(256, 128)
print(layer.weight.std().item())   # approximately sqrt(2/256) ≈ 0.088

# Explicit initialisation methods
def init_weights(m):
    if isinstance(m, nn.Linear):
        # Xavier/Glorot — good for Tanh/Sigmoid activations
        nn.init.xavier_uniform_(m.weight)

        # He/Kaiming — good for ReLU-family activations (PyTorch default)
        # nn.init.kaiming_uniform_(m.weight, nonlinearity="relu")

        nn.init.zeros_(m.bias)

model = nn.Sequential(
    nn.Linear(784, 256), nn.ReLU(),
    nn.Linear(256, 128), nn.ReLU(),
    nn.Linear(128, 10),
)
model.apply(init_weights)   # applies init_weights to every sub-module

# Why initialisation matters: too small → vanishing activations
# too large → exploding activations, especially in deep nets
x = torch.randn(100, 784)
for layer in model:
    x = layer(x)
    if hasattr(layer, "weight"):
        print(f"{layer}: activation std={x.std().item():.4f}")
# With good init, std should stay roughly stable across layers

# Custom initialisation from scratch
with torch.no_grad():
    layer.weight.normal_(mean=0.0, std=0.02)  # common for transformer init
    layer.bias.zero__()

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.