Hypothesis Testing Fundamentals

Statistical Inference & Applied Statistics

Chapter 4 · Hypothesis Testing Fundamentals

Chapter 2 ended on an unresolved tension: a z-score of 2.25 was called "notably large," but the chapter admitted it didn't yet have "the formal tool to make that determination with real statistical rigor." This chapter is that tool.

The Null and Alternative Hypothesis

Two competing claims

H₀ (null hypothesis): the default, "nothing changed" claim.
H₁ (alternative hypothesis): the claim that there genuinely is an effect or difference.

A hypothesis test never proves H₁ — it only asks whether the observed data provides strong enough evidence to reject H₀. Failing to reject H₀ isn't the same as proving it true; it just means the evidence wasn't strong enough to rule it out.

Significance Level — Directly Tied to Chapter 3's Confidence Level

The significance level (α) is the threshold risk of a false rejection a team is willing to accept — conventionally α = 0.05 (5%). It's not a new idea: α = 1 − confidence level, exactly Chapter 3's own 95% confidence level restated from the opposite direction.

The Test Statistic and P-Value

Z-statistic and p-value (known σ)

z = (x̄ − μ₀) / SE p-value = 2 × (1 − Φ(|z|)) for a two-tailed test, using the same standard normal CDF Φ from Probability & Statistics Fundamentals Chapter 8.

The p-value answers a precise question: if H₀ were actually true, how likely is a result at least this extreme, purely from sampling error? A small p-value means the observed data would be a genuinely unusual coincidence under H₀ — evidence favoring H₁ instead.

Decision rule

If p-value < α, reject H₀ ("statistically significant"). Otherwise, fail to reject H₀.

Resolving Chapter 2's Own Two Examples

Testing H₀: μ = 200ms against H₁: μ ≠ 200ms, reusing Chapter 2's own two scenarios:

Scenario	z	p-value	Decision at α=0.05
x̄=204ms, n=25, σ=30 (Ch.2's main example)	0.667	0.505	Fail to reject H₀ — confirms Ch.2's "unremarkable" call
90-unit shift, n=25, σ=200 (Ch.2's Exercise 1)	2.25	0.024	Reject H₀ — formally confirms Ch.2's "notably large" call

Chapter 2 could only eyeball these results informally. This chapter turns that instinct into a precise, defensible number.

Type I and Type II Errors

Every hypothesis test can go wrong in two distinct ways:

	H₀ is actually true	H₀ is actually false
Reject H₀	Type I error (false positive) — rate controlled directly by α	Correct decision
Fail to reject H₀	Correct decision	Type II error (false negative) — a real effect gets missed

Lowering α reduces Type I errors directly, but — with a fixed sample size — tends to increase Type II errors, since a stricter threshold makes real effects harder to detect too. Neither error can be eliminated without more data.

Two Real Cautions

The multiple-testing trap

At α = 0.05, testing 15 genuinely null hypotheses (no real effect in any of them) still produces, on average, 15 × 0.05 = 0.75 "statistically significant" results purely by chance. Running many tests and reporting only the "significant" ones — without correcting for how many tests were actually run — is a direct path to false conclusions, exactly the base-rate reasoning Probability & Statistics Fundamentals' own Bayes' Theorem chapter warned about from a different angle.

Statistical significance ≠ practical significance

With a large enough sample, even a genuinely tiny, meaningless difference can become "statistically significant" — a huge n shrinks the standard error enough that even trivial effects produce a small p-value. A significant result answers "is there probably a real effect," not "is this effect big enough to matter." Both questions need answering separately.

Hypothesis Testing in Code

import math

def standard_normal_cdf(z):
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))

def z_test(sample_mean, mu0, sigma, n):
    se = sigma / math.sqrt(n)
    z = (sample_mean - mu0) / se
    p_value = 2 * (1 - standard_normal_cdf(abs(z)))
    return z, p_value

z, p = z_test(sample_mean=204, mu0=200, sigma=30, n=25)
print(z, p)  # 0.667, 0.505 — matches Chapter 2's own "unremarkable" result

alpha = 0.05
print("reject H0" if p < alpha else "fail to reject H0")

Hands-On Exercises

Exercise 1

Testing H₀: μ = 50 against H₁: μ ≠ 50, a sample of n = 36 gives x̄ = 54, with known σ = 12. Compute the standard error, the z-statistic, and the p-value. State the decision at α = 0.05.

📄 View solution

Exercise 2

A team runs a hypothesis test and rejects H₀, concluding their new feature improved conversion. It later turns out the feature had no real effect at all — the observed difference was just sampling noise. Which type of error occurred: Type I or Type II? Now describe a second, different scenario where the opposite error type occurs instead.

📄 View solution

Exercise 3

A team runs 20 independent A/B tests in a single quarter, all at α = 0.05, and none of the 20 features actually has any real effect. Using this chapter's own multiple-testing finding, compute the expected number of tests that will show a "statistically significant" result purely by chance, and explain what this implies for how the team should interpret a report claiming "3 out of 20 tests were significant this quarter."

📄 View solution

Chapter 4 Quick Reference

H₀ (null, "no effect") vs H₁ (alternative, "real effect") — a test can reject H₀, never prove H₁
Significance level α = 1 − confidence level (Chapter 3) — conventionally 0.05
Z-statistic: z = (x̄ − μ₀)/SE; p-value: 2(1 − Φ(|z|)) — reject H₀ when p-value < α
Type I error: false positive (rejecting a true H₀), rate = α; Type II error: false negative (missing a real effect)
Running many tests inflates the expected number of false positives purely by chance — the multiple-testing trap
Statistical significance ("probably a real effect") is not the same question as practical significance ("is it big enough to matter")
Next chapter: The t-test and comparing two groups