What are learning rate schedulers in PyTorch and how do you use them?

A learning rate (LR) scheduler adjusts the learning rate during training. Starting with a high LR enables fast early progress; decaying it later allows finer convergence. PyTorch provides many schedulers in torch.optim.lr_scheduler.

import torch, torch.nn as nn, torch.optim as optim
from torch.optim import lr_scheduler

model = nn.Linear(10, 1)
optimizer = optim.AdamW(model.parameters(), lr=1e-3)

# CosineAnnealingLR — most popular modern choice
scheduler_cos = lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=100, eta_min=1e-6
)

# OneCycleLR — great for fast training
scheduler_1c = lr_scheduler.OneCycleLR(
    optimizer,
    max_lr=1e-2,
    steps_per_epoch=100,   # batches per epoch
    epochs=10,
)

# ReduceLROnPlateau — metric-driven
scheduler_plat = lr_scheduler.ReduceLROnPlateau(
    optimizer, mode="min", factor=0.5, patience=5, verbose=True
)

# Standard usage in training loop
for epoch in range(100):
    train_one_epoch(model, optimizer)   # forward + backward + step

    # --- Epoch-based schedulers ---
    scheduler_cos.step()    # call AFTER optimizer.step()

    # --- Metric-based scheduler ---
    val_loss = validate(model)
    scheduler_plat.step(val_loss)

    # --- OneCycleLR is per-batch ---
    # for batch in dataloader:
    #     optimizer.step()
    #     scheduler_1c.step()

    print(f"lr: {optimizer.param_groups[0]['lr']:.6f}")

Key rule: call scheduler.step() after optimizer.step(). For OneCycleLR and other per-batch schedulers, call scheduler.step() inside the batch loop, not the epoch loop.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.