Least Squares

Precise definition

For an overdetermined system $Ax\approx b$, a least-squares solution minimizes $\|Ax-b\|_2^2$. At a minimizer, the residual $r=b-A\hat x$ is orthogonal to every column of $A$, giving the normal equations $A^TA\hat x=A^Tb$.

Notation and mathematical language

If $A$ has independent columns, $A^TA$ is invertible and the unique minimizer is $(A^TA)^{-1}A^Tb$. Geometrically, $A\hat x$ is the orthogonal projection of $b$ onto $\operatorname{Col}A$. QR solves $R\hat x=Q^Tb$ without forming $A^TA$.

Conceptual picture

Least squares does not make inconsistent equations true; it chooses the attainable vector closest to the data in Euclidean distance. The residual contains the unexplained component perpendicular to the model subspace.

Conditions and key results

The method depends on the chosen norm and scaling. Ordinary least squares assumptions are needed for standard statistical uncertainty claims, not for the algebraic minimization itself. Correlation or a fitted line does not establish causation.

A reliable strategy

1. Build the design matrix with rows matched correctly to observations and inspect column rank.
2. Form the orthogonality condition or use QR; solve for coefficients without unnecessary explicit inversion.
3. Compute fitted values and residuals, then verify $A^Tr=0$.
4. Interpret coefficients, units, residual pattern, and model limitations separately from causal claims.

Fully worked example

Interpretation and application

Least squares underlies regression, calibration, imaging, and data fitting. A small residual measures in-sample geometric fit; it does not prove correct functional form, reliable extrapolation, independent errors, or a causal relationship.

Common mistakes

Verification and reasonableness

• Compute $A^T(b-A\hat x)$ and obtain zero within numerical tolerance.
• Compare residual sum of squares with simple alternative coefficients.
• Plot residuals versus predictors to look for systematic structure the model misses.

Practice

1. What equation characterizes a least-squares residual?
2. What geometric object is $A\hat x$?
3. Does least squares prove causation?

Further deduction

Forming $A^TA$ squares the spectral condition number when $A$ has full column rank, potentially worsening rounding error. QR factorization preserves the least-squares geometry more directly, and singular value decomposition handles rank deficiency while identifying a minimum-norm solution. Exact classroom algebra and reliable numerical implementation therefore may use different computational routes.

With an intercept in simple linear regression, the fitted line passes through $(\bar x,\bar y)$. The normal equations make residuals orthogonal to the all-ones column and the centred predictor, giving zero residual sum and zero residual–predictor cross-product. These are algebraic properties of the fit, not evidence that the linear model or causal story is correct.

Weighted least squares minimizes $\sum_iw_i(y_i-a_i^Tx)^2$ with positive weights, giving normal equations $A^TWAx=A^TWy$. Weights can reflect differing measurement precision, but arbitrary choices change the estimand and require explanation rather than being treated as cosmetic settings.

Related topics

• Orthogonal Projection
• Gram-Schmidt Process
• Linear Regression