What is the mathematical difference between L1 (Lasso) and L2 (Ridge) regularization, and why does L1 produce sparse solutions?

Both methods add a penalty term to the loss function to discourage large coefficients. Ridge (L2) adds λΣβᵢ²; Lasso (L1) adds λΣ|βᵢ|. The mathematical consequence of this difference is profound: L1 regularization can drive coefficients to exactly zero (producing sparse, interpretable models with automatic feature selection), while L2 shrinks coefficients toward zero but almost never makes them exactly zero.

The geometric explanation: think of the regularization term as a constraint region — L2's constraint Σβᵢ² ≤ t is a smooth circle/sphere, while L1's constraint Σ|βᵢ| ≤ t is a diamond/polytope with sharp corners on the axes. When you find the point where the elliptical loss contours first touch the constraint region, the circular L2 region rarely touches exactly on an axis (rarely zeroing a coefficient), but the diamond-shaped L1 region has corners precisely on the axes — the loss contours are statistically much more likely to first touch at one of these corners, zeroing out one or more coefficients.

from sklearn.linear_model import Ridge, Lasso
import numpy as np

# Ridge: shrinks coefficients but rarely to exactly zero
ridge = Ridge(alpha=1.0).fit(X_train, y_train)
print('Ridge coefficients:', ridge.coef_)
# e.g. [0.31, 0.02, 0.18, 0.04]  — small but non-zero

# Lasso: drives some coefficients to exactly zero
lasso = Lasso(alpha=0.1).fit(X_train, y_train)
print('Lasso coefficients:', lasso.coef_)
# e.g. [0.45, 0.0, 0.22, 0.0]   — automatic feature selection!

selected_features = np.where(lasso.coef_ != 0)[0]
print(f'Lasso selected {len(selected_features)} of {len(lasso.coef_)} features')

# ElasticNet: combines both penalties
from sklearn.linear_model import ElasticNet
en = ElasticNet(alpha=0.1, l1_ratio=0.5).fit(X_train, y_train)

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.