Python / PyTorch Fundamentals Interview Questions
What are learning rate schedulers in PyTorch and how do you use them?
A learning rate scheduler adjusts the learning rate during training — typically starting high for fast initial progress and decaying for fine-grained convergence. Schedulers wrap an optimizer and must be stepped after each epoch (or each batch for some schedulers).
| Scheduler | Behaviour | Step |
|---|---|---|
| StepLR | Multiply lr by gamma every step_size epochs | Per epoch |
| MultiStepLR | Decay at specified milestone epochs | Per epoch |
| ExponentialLR | lr *= gamma every epoch | Per epoch |
| CosineAnnealingLR | Cosine decay from lr to eta_min | Per epoch |
| OneCycleLR | Warmup then cosine decay (superconvergence) | Per batch |
| ReduceLROnPlateau | Reduce lr when metric stops improving | Per epoch (with metric |
| CosineAnnealingWarmRestarts | Cosine with periodic restarts | Per epoch |
import torch, torch.optim as optim import torch.nn as nn model = nn.Linear(128, 10) optimizer = optim.SGD(model.parameters(), lr=0.1) # ââ StepLR: multiply lr by 0.1 every 30 epochs scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1) # ââ CosineAnnealingLR: smooth cosine decay scheduler = optim.lr_scheduler.CosineAnnealingLR( optimizer, T_max=100, eta_min=1e-6 ) # ââ OneCycleLR: requires total_steps at init scheduler = optim.lr_scheduler.OneCycleLR( optimizer, max_lr=0.1, total_steps=100 * len([1]*1000), # epochs * batches_per_epoch ) # ââ ReduceLROnPlateau: triggered by validation loss scheduler = optim.lr_scheduler.ReduceLROnPlateau( optimizer, mode="min", factor=0.5, patience=5, verbose=True ) # ââ Training loop integration for epoch in range(100): train_loss = 0.0 # ... train ... val_loss = 0.0 # ... validate ... # Most schedulers: step after epoch scheduler.step() # for StepLR, CosineAnnealingLR etc. # scheduler.step(val_loss) # for ReduceLROnPlateau (needs metric) # Check current lr current_lr = optimizer.param_groups[0]["lr"] print(f"Epoch {epoch}: lr={current_lr:.6f}")
When should you call scheduler.step() for most epoch-based learning rate schedulers?
Which PyTorch scheduler is most suitable when you want to reduce learning rate only if validation loss stops improving?
More Related questions...
