Explain backpropagation mathematically. How does the chain rule enable computing gradients through many layers?

Backpropagation is the algorithm for computing the gradient of a scalar loss L with respect to every parameter in the network. It exploits the chain rule of calculus: if the loss depends on parameter W through intermediate quantities h₁, h₂, ..., hₙ, then ∂L/∂W = (∂L/∂hₙ)(∂hₙ/∂hₙ₋₁)···(∂h₁/∂W). Backprop applies the chain rule systematically starting from the loss and working backwards through each layer, accumulating local gradients.

At each layer, two quantities are needed: the local gradient (how does the layer's output change with its input/weights?) and the upstream gradient (how does the loss change with this layer's output?). Multiplying them gives the gradient flowing to the layer's parameters and to its input, which becomes the upstream gradient for the preceding layer.

import torch import torch.nn as nn model = nn.Sequential( nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1) ) loss_fn = nn.MSELoss() optimizer = torch.optim.SGD(model.parameters(), lr=0.01) x = torch.randn(32, 10) y = torch.randn(32, 1) # --- Standard training step --- optimizer.zero_grad() # 1. Clear old gradients (they accumulate!) y_hat = model(x) # 2. Forward pass — build computation graph loss = loss_fn(y_hat, y)# 3. Compute scalar loss loss.backward() # 4. Backprop — traverse graph in reverse # populates .grad for every parameter optimizer.step() # 5. Update parameters: W -= lr * W.grad # Inspect gradients of first layer print(model[0].weight.grad.shape) # torch.Size([64, 10]) # Manual chain rule for a single neuron: # loss = (y_hat - y)^2, y_hat = w*x + b # dL/dw = 2*(y_hat - y) * x <- upstream * local w = torch.tensor([2.0], requires_grad=True) x_s = torch.tensor([3.0]) y_s = torch.tensor([1.0]) loss_s = (w * x_s - y_s) ** 2 loss_s.backward() print(w.grad) # tensor([40.]) == 2*(2*3-1)*3

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.