Compare SGD, SGD with momentum, RMSProp, and Adam optimizers. When do you choose each?

All these optimizers share the same goal — updating parameters to reduce loss — but differ in how they use gradient history to adapt the update step. Understanding the mechanics helps diagnose slow training and poor generalisation.

import torch
import torch.nn as nn

model = nn.Linear(10, 1)

# SGD — baseline, works but needs careful lr tuning
opt_sgd = torch.optim.SGD(model.parameters(), lr=0.01)

# SGD + Momentum — adds velocity; β=0.9 is standard
opt_mom = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9,
                           weight_decay=1e-4)  # L2 regularisation

# Adam — adaptive learning rate + momentum; best default for DL
opt_adam = torch.optim.Adam(model.parameters(),
                             lr=1e-3,      # default, usually works
                             betas=(0.9, 0.999),  # momentum terms
                             eps=1e-8,
                             weight_decay=1e-5)

# AdamW — Adam with decoupled weight decay (better than Adam + L2)
opt_adamw = torch.optim.AdamW(model.parameters(), lr=1e-3,
                               weight_decay=1e-2)

# Learning rate schedulers — change lr during training
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(opt_adam, T_max=100)
for epoch in range(100):
    # ... training loop ...
    scheduler.step()  # decrease lr following cosine curve

When to choose: Adam is the safe default for most deep learning tasks. SGD with momentum often achieves better final generalisation on image classification tasks (the finding that motivated the NLP community's shift back to AdamW for fine-tuning pre-trained transformers). AdamW is now the standard for fine-tuning large language models.