Hypothesis testing

A courtroom presumes innocence; a researcher presumes no effect. Both then ask whether the evidence is strong enough to reject that starting assumption — and both can convict an innocent (Type I) or free the guilty (Type II). That analogy unlocks most hypothesis-testing questions on the NCE.

You always test two competing claims. The null hypothesis (H0) says there is no difference / no relationship — the status quo. The alternative hypothesis (H1) says there is an effect. You never prove H1; you either reject or fail to reject the null.

The decision hinges on the p-value — the probability of getting your result (or more extreme) if the null were true. Compare it to alpha (α), the significance level you set in advance (conventionally .05). If p < α, the result is statistically significant and you reject H0. If p ≥ α, you fail to reject. Note what p is not: it is not the probability the null is true, and not the size of the effect.

Now the two errors examiners drill. A Type I error is rejecting a true null — a false positive (you 'found' an effect that isn't real). Its probability equals alpha. A Type II error is failing to reject a false null — a false negative (you missed a real effect). Its probability is beta (β), and power = 1 − β (the ability to detect a true effect). Mnemonic: Type I = 1 = false positive (α); Type II = 2 = false negative (β). Lowering alpha reduces Type I risk but raises Type II risk — a trade-off.

Finally, statistical vs. practical significance. A tiny, trivial effect can be statistically significant with a large enough sample. 'Significant' means unlikely due to chance — not 'important' or 'clinically meaningful.'

Clinical Pearl: Type I = false positive (reject a true null, probability = α); Type II = false negative (miss a real effect, probability = β). And statistical significance ≠ practical importance — a big enough sample can make trivial effects 'significant.'