Python / PyTorch Fundamentals Interview Questions
What are learning rate schedulers in PyTorch and how do you use them?
A learning rate (LR) scheduler adjusts the learning rate during training. Starting with a high LR enables fast early progress; decaying it later allows finer convergence. PyTorch provides many schedulers in torch.optim.lr_scheduler.
| Scheduler | Behaviour | Use case |
|---|---|---|
| StepLR | Multiply lr by gamma every step_size epochs | Simple decay; quick experiments |
| MultiStepLR | Decay at specific epoch milestones | ResNet training schedules |
| CosineAnnealingLR | Cosine curve from lr to eta_min | Most modern training runs |
| OneCycleLR | Warmup to max_lr then cosine decay | Super-convergence; fast training |
| ReduceLROnPlateau | Reduce lr when metric stops improving | When training time is unknown |
| LinearLR | Linear warm-up | Transformer fine-tuning |
| CosineAnnealingWarmRestarts | Cosine + periodic restarts (SGDR) | Ensemble-style training |
import torch, torch.nn as nn, torch.optim as optim from torch.optim import lr_scheduler model = nn.Linear(10, 1) optimizer = optim.AdamW(model.parameters(), lr=1e-3) # CosineAnnealingLR â most popular modern choice scheduler_cos = lr_scheduler.CosineAnnealingLR( optimizer, T_max=100, eta_min=1e-6 ) # OneCycleLR â great for fast training scheduler_1c = lr_scheduler.OneCycleLR( optimizer, max_lr=1e-2, steps_per_epoch=100, # batches per epoch epochs=10, ) # ReduceLROnPlateau â metric-driven scheduler_plat = lr_scheduler.ReduceLROnPlateau( optimizer, mode="min", factor=0.5, patience=5, verbose=True ) # Standard usage in training loop for epoch in range(100): train_one_epoch(model, optimizer) # forward + backward + step # --- Epoch-based schedulers --- scheduler_cos.step() # call AFTER optimizer.step() # --- Metric-based scheduler --- val_loss = validate(model) scheduler_plat.step(val_loss) # --- OneCycleLR is per-batch --- # for batch in dataloader: # optimizer.step() # scheduler_1c.step() print(f"lr: {optimizer.param_groups[0]['lr']:.6f}")
Key rule: call scheduler.step() after optimizer.step(). For OneCycleLR and other per-batch schedulers, call scheduler.step() inside the batch loop, not the epoch loop.
When should scheduler.step() be called relative to optimizer.step()?
Which scheduler is well-suited when you don't know how many epochs you'll train for?
More Related questions...
