Least Squares

A rigorous guide to inconsistent systems, orthogonal residuals, normal equations, QR, and interpretation.

Orthogonal projection separates a vector into fitted and perpendicular components. A vector is decomposed into a component along a subspace and a right-angled residual. Basis arrows show how coordinates reconstruct the projected result.

Precise definition

For an overdetermined system $Ax\approx b$, a least-squares solution minimizes $\|Ax-b\|_2^2$. At a minimizer, the residual $r=b-A\hat x$ is orthogonal to every column of $A$, giving the normal equations $A^TA\hat x=A^Tb$.

Notation and mathematical language

If $A$ has independent columns, $A^TA$ is invertible and the unique minimizer is $(A^TA)^{-1}A^Tb$. Geometrically, $A\hat x$ is the orthogonal projection of $b$ onto $\operatorname{Col}A$. QR solves $R\hat x=Q^Tb$ without forming $A^TA$.

Conceptual picture

Least squares does not make inconsistent equations true; it chooses the attainable vector closest to the data in Euclidean distance. The residual contains the unexplained component perpendicular to the model subspace.

Fully worked example

Interpretation and application

Least squares underlies regression, calibration, imaging, and data fitting. A small residual measures in-sample geometric fit; it does not prove correct functional form, reliable extrapolation, independent errors, or a causal relationship.