Linear Regression

A regression line summarizes an average linear pattern; residuals show what the line misses.

Model form

A fitted linear model is written

where $\hat y$ is the predicted response, $b$ is slope, and $a$ is intercept. The hat distinguishes a prediction from an observed value $y$.

The explanatory variable is $x$ and the response variable is $y$.

Least-squares criterion

For each observation, the residual is

Least-squares regression chooses $a$ and $b$ to minimize

Squaring prevents positive and negative residuals from cancelling and penalizes large errors.

Interpreting slope

The slope describes predicted response change for one additional unit of the explanatory variable.

Always include both variable units.

Interpreting the intercept

The intercept predicts $y$ when $x=0$. It is meaningful only if zero is realistic and reasonably near the observed range.

An intercept can be mathematically necessary for the line while lacking a sensible contextual interpretation.

Worked residual

Using $\hat y=4.2x+58$, a student with $x=5$ study hours has predicted score

If the observed score is $82$, then

A positive residual means the observation lies above the fitted line.

Residual plots

Plot residuals against fitted values or $x$. A good linear model has residuals scattered around zero without a clear curve, funnel shape, or time pattern.

Curvature suggests nonlinearity; changing spread suggests nonconstant variability; isolated large residuals suggest outliers.

Coefficient of determination

$R^2$ is the proportion of response variation explained by the fitted model in the dataset. In simple linear regression with an intercept,

An $R^2$ of $0.64$ means $64\%$ of observed response variation is accounted for by the linear model—not that predictions are $64\%$ accurate or that causation is $64\%$ proven.

Interpolation and extrapolation

Interpolation predicts within the observed explanatory range. Extrapolation predicts beyond it and is riskier because the relationship may change.

Even interpolation inherits uncertainty and any bias in the data.

Influential observations

Points with unusual $x$-values can strongly affect slope and intercept. Examine leverage, residual size, and data quality. Report analyses with and without a point only when justified.

Do not remove valid data because they weaken a desired model.

Association versus causation

Regression estimates a conditional association. Causal interpretation requires appropriate design and assumptions, often including random assignment, controlled conditions, and attention to confounding.

A precise fitted line cannot repair biased sampling or poor measurement.

Making a prediction

Substitute an allowed $x$ into the model, report $\hat y$ with units and sensible precision, and state whether the prediction is within the data range.

When available, prediction intervals communicate uncertainty more honestly than a single point estimate.

Common mistakes

Interpreting slope without units or context. Name both variables.

Forcing meaning onto an unrealistic intercept. Check whether $x=0$ makes sense.

Calculating residual as predicted minus observed. Use observed minus predicted.

Calling $R^2$ prediction accuracy or causal strength. It summarizes variation explained in the sample.

Extrapolating far beyond the data without warning. Model form may not persist.

Quick self-check

• Are explanatory and response variables identified correctly?
• Does the scatter plot support a linear model?
• Is slope interpreted as predicted change with units?
• Are residuals patternless around zero?
• Is $R^2$ described accurately?
• Are outliers, extrapolation, uncertainty, bias, and causal limits addressed?

Related topics

• Correlation
• Scatter Plots
• Confidence Intervals
• Bias in Data