Hypothesis Testing

Precise definition

A hypothesis test compares data with a null model $H_0$ using a pre-specified statistic. The p-value is the probability, assuming $H_0$ and all test-model assumptions, of obtaining a statistic at least as incompatible with $H_0$ as the observed value. It is not the probability that $H_0$ is true.

Notation and mathematical language

A significance level $\alpha$ controls the long-run Type I error rate for the defined procedure under $H_0$. Type II error is failing to reject when a specified alternative is true; power is one minus that probability. One- and two-sided alternatives define different extremeness regions and must be chosen before seeing results.

Conceptual picture

Testing calibrates surprise under a model. A small p-value indicates tension between data and the null package, which includes sampling, independence, distribution, and analysis choices. Effect size and confidence interval answer magnitude questions that a threshold decision cannot.

Conditions and key results

Valid calibration requires the test's design and assumptions: random sampling or randomization where invoked, independence or correct dependence model, appropriate distribution or large-sample approximation, and no undisclosed multiple testing. Statistical significance neither proves practical importance nor causation.

A reliable strategy

1. State population parameter, $H_0$, $H_A$, direction, and $\alpha$ before inspecting the test result.
2. Check design and distributional conditions and select the test statistic and null distribution.
3. Compute the observed statistic and tail probability or compare with a critical region.
4. Report reject/fail-to-reject language, effect estimate, interval, assumptions, multiplicity, and the limits of causal interpretation.

Fully worked example

Interpretation and application

Tests support experiments, quality control, and observational studies. Random assignment can justify causal comparisons under compliance and design conditions; a low p-value from observational association does not remove confounding.

Common mistakes

Verification and reasonableness

• Recompute the statistic from the raw summary and check tail direction.
• Compare the p-value conclusion with a confidence interval for the same parameter.
• Audit assumptions, exclusions, stopping rules, and number of tested hypotheses.

Practice

1. What does a p-value condition on?
2. What is Type I error?
3. Does $p>0.05$ prove equality?

Further deduction

Multiple testing inflates the chance of at least one false rejection. Testing 20 independent true nulls at $\alpha=0.05$ gives family-wise error $1-0.95^{20}\approx0.642$, not 0.05. Bonferroni controls family-wise error by comparing each p-value with $\alpha/m$; false-discovery-rate procedures target a different error criterion. The planned family of hypotheses must be defined.

Power depends on effect size, sample size, variability, significance threshold, and test direction. Planning a sample size requires a scientifically meaningful alternative, not only a desire for $p<0.05$. Very large samples can detect negligible differences; a confidence interval keeps the plausible effect magnitudes visible and supports practical interpretation.

Related topics

• Confidence Intervals
• Sampling
• Normal Distribution