Python / PyTorch Fundamentals Interview Questions
What is autograd in PyTorch and how does it compute gradients?
PyTorch's autograd engine implements automatic differentiation. When you perform operations on tensors with requires_grad=True, PyTorch records every operation in a dynamic computation graph. Calling .backward() on a scalar loss traverses this graph in reverse using the chain rule, accumulating gradients in each tensor's .grad attribute.
import torch # requires_grad=True tells PyTorch to track this tensor x = torch.tensor([2.0, 3.0], requires_grad=True) # Forward pass â operations are recorded y = x ** 2 # y = [4.0, 9.0] z = y.sum() # z = 13.0 (scalar) # Backward pass â computes dz/dx using chain rule z.backward() print(x.grad) # tensor([4., 6.]) dz/dx = 2x # Verify: dz/d(x[0]) = d(x[0]^2)/d(x[0]) = 2*x[0] = 4 â # Gradients ACCUMULATE â always zero before next backward! x.grad.zero_() # or optimizer.zero_grad() # Non-leaf tensors (created by ops) have grad_fn a = torch.tensor(3.0, requires_grad=True) b = a * 2 print(b.grad_fn) # <MulBackward0 object> print(b.requires_grad) # True â inherited from a # Detach: stop tracking a tensor c = b.detach() # c shares data with b but no grad history print(c.requires_grad) # False # torch.no_grad(): context manager to disable gradient tracking with torch.no_grad(): inference = a * 2 # faster, no graph built print(inference.requires_grad) # False
| Concept | What it is |
|---|---|
| requires_grad=True | Tells autograd to track operations on this tensor |
| .grad | Accumulated gradient after .backward() lives on leaf tensors |
| grad_fn | Reference to the function that created a non-leaf tensor |
| .backward() | Traverses graph backwards, fills .grad via chain rule |
| .detach() | Returns tensor with same data but no gradient history |
| torch.no_grad() | Context: disables gradient tracking (inference, validation) |
What method do you call to trigger gradient computation in PyTorch?
Why must you call optimizer.zero_grad() (or tensor.grad.zero_()) before each backward pass?
More Related questions...
