How does Dropout work mathematically, and why does it act as regularization?

During training, Dropout randomly sets each neuron's output to zero with probability p (the drop probability) and scales the remaining activations by 1/(1-p) to preserve the expected sum. This means each forward pass trains a different thinned sub-network — with n neurons, there are 2ⁿ possible sub-networks, and each training step updates a random one.

The regularisation effect comes from several mechanisms: (1) it prevents co-adaptation — neurons cannot rely on specific other neurons always being present, so each must learn useful features independently; (2) it is mathematically equivalent to training an exponentially large ensemble and averaging their predictions at test time (where Dropout is disabled); (3) the multiplicative noise acts similarly to L2 regularisation on the weights. At inference, Dropout is disabled and all neurons are active — the 1/(1-p) scaling during training ensures the expected value of each neuron's output is the same during training and inference.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 512), nn.ReLU(),
    nn.Dropout(p=0.5),              # drop 50% of neurons
    nn.Linear(512, 256), nn.ReLU(),
    nn.Dropout(p=0.3),
    nn.Linear(256, 10)
)

# Training: Dropout is ACTIVE (neurons randomly zeroed)
model.train()
x = torch.ones(1, 784)
out1 = model(x)
out2 = model(x)  # different result! different neurons dropped each time

# Inference: Dropout is DISABLED (all neurons active)
model.eval()
with torch.no_grad():
    out3 = model(x)
    out4 = model(x)  # same result — deterministic

# Inverted Dropout (PyTorch default):
# Scale by 1/(1-p) DURING training, not during inference
# => test-time output has correct expected value without scaling
dp = nn.Dropout(p=0.5)
model.train()
x_in = torch.ones(10)
print(dp(x_in))  # ~5 zeros, remaining values are 2.0 (scaled by 1/0.5)