Explain the mathematical foundation of PCA. What do eigenvectors and eigenvalues represent in this context?

Principal Component Analysis (PCA) finds a new orthogonal coordinate system where the axes (principal components) are ordered by the amount of variance they capture in the data. Mathematically, PCA computes the eigenvectors and eigenvalues of the data's covariance matrix Σ = (1/n)XᵀX (after centering X to zero mean).

Each eigenvector of the covariance matrix points in a direction in the original feature space; the corresponding eigenvalue equals the variance of the data when projected onto that eigenvector's direction. The eigenvector with the largest eigenvalue is the first principal component — the direction of maximum variance in the data. PCA sorts eigenvectors by descending eigenvalue and keeps the top k to reduce dimensionality while retaining as much variance (information) as possible.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# ALWAYS standardize before PCA — PCA is scale-sensitive
X_scaled = StandardScaler().fit_transform(X)

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print('Explained variance ratio:', pca.explained_variance_ratio_)
# e.g. [0.45, 0.23] — first PC explains 45% of variance, second 23%

print('Principal components (eigenvectors):', pca.components_)
print('Eigenvalues:', pca.explained_variance_)

# Manual computation via covariance matrix eigendecomposition
cov_matrix = np.cov(X_scaled, rowvar=False)
eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)
# Sort descending — eigh returns ascending order
idx = np.argsort(eigenvalues)[::-1]
eigenvalues, eigenvectors = eigenvalues[idx], eigenvectors[:, idx]

Choosing k: a common heuristic is to plot cumulative explained variance ratio and pick the smallest k that retains, say, 90-95% of total variance — this is exactly what PCA(n_components=0.95) automates in scikit-learn.