Explain the mathematical foundation of PCA. What do eigenvectors and eigenvalues represent in this context?

Principal Component Analysis (PCA) finds a new orthogonal coordinate system where the axes (principal components) are ordered by the amount of variance they capture in the data. Mathematically, PCA computes the eigenvectors and eigenvalues of the data's covariance matrix Σ = (1/n)XᵀX (after centering X to zero mean).

Each eigenvector of the covariance matrix points in a direction in the original feature space; the corresponding eigenvalue equals the variance of the data when projected onto that eigenvector's direction. The eigenvector with the largest eigenvalue is the first principal component — the direction of maximum variance in the data. PCA sorts eigenvectors by descending eigenvalue and keeps the top k to reduce dimensionality while retaining as much variance (information) as possible.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# ALWAYS standardize before PCA — PCA is scale-sensitive
X_scaled = StandardScaler().fit_transform(X)

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print('Explained variance ratio:', pca.explained_variance_ratio_)
# e.g. [0.45, 0.23] — first PC explains 45% of variance, second 23%

print('Principal components (eigenvectors):', pca.components_)
print('Eigenvalues:', pca.explained_variance_)

# Manual computation via covariance matrix eigendecomposition
cov_matrix = np.cov(X_scaled, rowvar=False)
eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)
# Sort descending — eigh returns ascending order
idx = np.argsort(eigenvalues)[::-1]
eigenvalues, eigenvectors = eigenvalues[idx], eigenvectors[:, idx]

Choosing k: a common heuristic is to plot cumulative explained variance ratio and pick the smallest k that retains, say, 90-95% of total variance — this is exactly what PCA(n_components=0.95) automates in scikit-learn.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.