Why does convexity of the loss function matter for optimization algorithms like gradient descent, mathematically?

A function is convex if a line segment connecting any two points on its graph lies above (or on) the graph itself — equivalently, its second derivative (or Hessian, in multiple dimensions) is non-negative everywhere. The critical property of a convex function is that any local minimum is also the global minimum — there are no other low points the optimisation could get stuck in.

This is why gradient descent on convex losses like OLS's squared error or logistic regression's cross-entropy is guaranteed to converge to the globally optimal solution (given a small enough learning rate), regardless of where the parameters are initialised. Non-convex loss landscapes — like those of neural networks — can have many local minima and saddle points, meaning gradient descent's final result depends on initialisation and may not find the best possible solution; this is also why neural network training often relies on heuristics (different initialisations, momentum, learning rate schedules) that convex optimisation never needs.

import numpy as np

# Convex function: a simple parabola has exactly one minimum
def convex_loss(theta):
    return (theta - 3) ** 2 + 1

# Non-convex function: multiple local minima
def nonconvex_loss(theta):
    return np.sin(theta) * theta**0.5 if theta > 0 else theta**2

# Verify convexity numerically via second derivative sign
def second_derivative_check(f, x, h=1e-5):
    return (f(x + h) - 2 * f(x) + f(x - h)) / h**2

thetas = np.linspace(0.1, 10, 50)
second_derivs = [second_derivative_check(convex_loss, t) for t in thetas]
print('All non-negative (convex)?', all(d >= 0 for d in second_derivs))

# scikit-learn's LinearRegression, LogisticRegression, and Ridge/Lasso
# all use convex losses, so the solver's result is deterministic
# given the same data, regardless of how it's initialized internally