Why does convexity of the loss function matter for optimization algorithms like gradient descent, mathematically?

A function is convex if a line segment connecting any two points on its graph lies above (or on) the graph itself — equivalently, its second derivative (or Hessian, in multiple dimensions) is non-negative everywhere. The critical property of a convex function is that any local minimum is also the global minimum — there are no other low points the optimisation could get stuck in.

This is why gradient descent on convex losses like OLS's squared error or logistic regression's cross-entropy is guaranteed to converge to the globally optimal solution (given a small enough learning rate), regardless of where the parameters are initialised. Non-convex loss landscapes — like those of neural networks — can have many local minima and saddle points, meaning gradient descent's final result depends on initialisation and may not find the best possible solution; this is also why neural network training often relies on heuristics (different initialisations, momentum, learning rate schedules) that convex optimisation never needs.

import numpy as np

# Convex function: a simple parabola has exactly one minimum
def convex_loss(theta):
    return (theta - 3) ** 2 + 1

# Non-convex function: multiple local minima
def nonconvex_loss(theta):
    return np.sin(theta) * theta**0.5 if theta > 0 else theta**2

# Verify convexity numerically via second derivative sign
def second_derivative_check(f, x, h=1e-5):
    return (f(x + h) - 2 * f(x) + f(x - h)) / h**2

thetas = np.linspace(0.1, 10, 50)
second_derivs = [second_derivative_check(convex_loss, t) for t in thetas]
print('All non-negative (convex)?', all(d >= 0 for d in second_derivs))

# scikit-learn's LinearRegression, LogisticRegression, and Ridge/Lasso
# all use convex losses, so the solver's result is deterministic
# given the same data, regardless of how it's initialized internally

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.