Project data onto informative directions
PCA finds orthogonal directions that capture decreasing amounts of variance, then projects the data onto those directions.
The idea
Rotate your viewpoint so the direction with the most visible spread becomes the first axis, then summarize the data along that direction.
After centering (and often scaling where appropriate), PCA uses covariance/eigenstructure or an equivalent SVD formulation to obtain principal directions and explained variance.
How it works
- Center the data; optionally standardize features when scale differences are not meaningful and would otherwise dominate.
- Find orthogonal directions of maximum variance using covariance eigenvectors or SVD.
- Order components by explained variance.
- Project samples onto the selected components and validate whether the reduced representation is useful for the downstream task.
Projection onto a principal direction
The first component chooses the unit direction v with maximum projected variance; later components are orthogonal to earlier ones.
z = X_centered v, with ||v||=1 chosen to maximize Var(X_centered v)
Check your understanding
Does ordinary PCA choose directions using the target label?
- No, PCA is unsupervised
- Yes, always
- Only for decision trees
- Only after sigmoid
Answer and explanation
Ordinary PCA is unsupervised: it chooses directions from feature variation, not from target labels.
A common misconception
PCA does not find the features most predictive of y. It finds directions of feature variance without using y.
Data & AI Foundations
Understand distributions, projections, evaluation, and explanations so model outputs are easier to reason about.