Python / Python Deep Learning and Neural Networks Interview Questions
How do you diagnose a neural network that is not training correctly from its loss curves?
Reading loss curves is one of the most important practical skills in deep learning. The shape of the training and validation loss over time reveals the failure mode and guides the fix.
| Loss curve shape | Diagnosis | Likely fix |
|---|---|---|
| Loss is NaN from the start | Exploding gradients or bad init | Gradient clipping, lower lr, check data for inf/NaN |
| Loss doesn't decrease at all | Vanishing gradient, lr too low, dead neurons | Check activations, raise lr, use He init + ReLU |
| Loss decreases then plateaus early | Learning rate too high or model too small | Reduce lr / lr schedule, increase capacity |
| Train loss low, val loss high (large gap) | Overfitting | More regularisation: dropout, weight decay, augmentation, early stopping |
| Both losses plateau at high value | Underfitting (high bias) | Increase model capacity, train longer, reduce regularisation |
| Loss oscillates wildly | Learning rate too high | Reduce lr, use lr schedule, check batch size |
import torch import torch.nn as nn # Checking for gradient issues model = nn.Linear(10, 5) optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) for step, (X, y) in enumerate(loader): optimizer.zero_grad() loss = criterion(model(X), y) loss.backward() # Check for NaN/Inf in loss if not torch.isfinite(loss): print(f'Step {step}: non-finite loss = {loss.item()}') break # Monitor gradient norms total_norm = 0 for p in model.parameters(): if p.grad is not None: total_norm += p.grad.data.norm(2).item() ** 2 total_norm = total_norm ** 0.5 if step % 100 == 0: print(f'Step {step}: loss={loss.item():.4f} grad_norm={total_norm:.4f}') optimizer.step() # Check dead ReLU neurons def count_dead_neurons(model, X): activations = [] def hook(m, inp, out): activations.append((out <= 0).float().mean().item()) handles = [l.register_forward_hook(hook) for l in model.modules() if isinstance(l, nn.ReLU)] with torch.no_grad(): model(X) for h in handles: h.remove() return activations # fraction of dead neurons per layer
If training loss decreases steadily but validation loss diverges upward, what is the most likely diagnosis?
What does a training loss that never decreases (stays near its initial value from epoch 1) typically indicate?
More Related questions...
