The t-test & Comparing Two Groups

Statistical Inference & Applied Statistics

Chapter 5 · The t-test & Comparing Two Groups

Every z-test in Chapter 4 quietly assumed the population standard deviation, σ, was already known. In real work, that's almost never true — only the sample standard deviation s is available, computed exactly the way Probability & Statistics Fundamentals' own Chapter 9 defined it, with Bessel's n−1 correction. That single substitution changes the math more than it might seem.

Why Estimating σ Changes the Distribution

When σ is estimated from the same limited sample as s, the resulting test statistic carries extra uncertainty on top of ordinary sampling error — and its true sampling distribution is no longer exactly normal. It follows the t-distribution instead: shaped like the normal bell curve, but with heavier tails, reflecting that extra layer of uncertainty from estimating s itself.

Degrees of freedom — directly from Chapter 9's own formula

A one-sample t-test uses df = n − 1 — exactly the denominator of the sample variance formula, Probability & Statistics Fundamentals Chapter 9's own Bessel's correction, reused here as the t-distribution's own shape parameter.

The t-distribution converges to normal as n grows

As df (and therefore n) grows, the t-distribution's tails thin out and it converges toward the standard normal — the extra uncertainty from estimating s matters less and less with more data, exactly the CLT's own "more data, more predictable" pattern from Chapter 8 of the sibling course, now applied to the act of estimating σ itself.

The One-Sample t-test

One-sample t-statistic

t = (x̄ − μ₀) / (s/√n) — identical in shape to Chapter 4's z-statistic, but built from the sample standard deviation s, and compared against a t-distribution with df = n − 1, not the standard normal.

Worked example: a team doesn't know the true σ, only their sample — n = 25, x̄ = 204ms, s = 32ms. Testing H₀: μ = 200:

Quantity	Value
SE = s/√n	32/5 = 6.4ms
t	(204−200)/6.4 = 0.625
df	25 − 1 = 24
Critical value (two-tailed, α=0.05, df=24)	2.064
Decision	\|0.625\| < 2.064 → fail to reject H₀

Same conclusion as Chapter 4's own z-test on similar numbers — unsurprising, since a t-distribution with 24 degrees of freedom is already close to normal.

Comparing Two Groups: The Two-Sample t-test

Comparing two independent groups' means — exactly the shape of an A/B test — uses Welch's t-test, which doesn't assume the two groups share the same variance (a more honest default than assuming equal variances, and the version most statistical software now uses by default):

Welch's two-sample t-statistic

t = (x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂)

Degrees of freedom for Welch's test — a deliberate simplification

Welch's test technically needs the Welch–Satterthwaite equation for its exact degrees of freedom — a genuinely messy formula. A common, more conservative simplification (used here) is df = min(n₁, n₂) − 1, which tends to slightly understate the true df and therefore errs toward being more cautious about rejecting H₀ — a reasonable simplification for this course's own scope.

Worked example: Group A (control), n₁=20, x̄₁=205ms, s₁=25ms; Group B (treatment), n₂=22, x̄₂=190ms, s₂=28ms.

Quantity	Value
SE_diff = √(25²/20 + 28²/22)	≈ 8.178
t = (205−190)/8.178	≈ 1.834
df = min(20,22) − 1	19
Critical value (two-tailed, α=0.05, df=19)	2.093
Decision	\|1.834\| < 2.093 → fail to reject H₀

A visible difference isn't automatically a significant one

The treatment group's response time looks noticeably lower — 190ms vs 205ms, a real 15ms, roughly 7% relative drop. Yet the t-test can't rule out that this gap is just sampling noise at this sample size. This is exactly the gap Chapter 6's own A/B testing chapter exists to close properly — eyeballing a difference and formally testing it are genuinely different things.

The t-test in Code

import math

# Common two-tailed alpha=0.05 critical values, by degrees of freedom
T_CRITICAL_05 = {9: 2.262, 14: 2.145, 15: 2.131, 19: 2.093, 24: 2.064, 30: 2.042}

def one_sample_t(sample_mean, mu0, s, n):
    se = s / math.sqrt(n)
    t = (sample_mean - mu0) / se
    return t, n - 1

def welch_two_sample_t(mean1, s1, n1, mean2, s2, n2):
    se_diff = math.sqrt((s1**2 / n1) + (s2**2 / n2))
    t = (mean1 - mean2) / se_diff
    df = min(n1, n2) - 1  # conservative simplification
    return t, df

t, df = one_sample_t(204, mu0=200, s=32, n=25)
print(t, df)  # 0.625, 24

t2, df2 = welch_two_sample_t(205, 25, 20, 190, 28, 22)
print(t2, df2)  # 1.834..., 19

Hands-On Exercises

Exercise 1

Testing H₀: μ = 50, a sample of n = 16 gives x̄ = 54, s = 10. Compute the standard error, the t-statistic, and the degrees of freedom. Using the critical value 2.131 (two-tailed, α=0.05, df=15), state the decision.

📄 View solution

Exercise 2

Group 1: n₁=15, x̄₁=48, s₁=6. Group 2: n₂=18, x̄₂=53, s₂=7. Compute Welch's t-statistic, the conservative degrees of freedom, and using the critical value 2.145 (two-tailed, α=0.05, df=14), state the decision.

📄 View solution

Exercise 3

Explain, in your own words, why the t-distribution needs heavier tails than the normal distribution when σ is unknown and estimated by s, and why a small-sample t-test's critical value (like 2.262 at df=9) is noticeably larger than the z-test's fixed 1.96. Then explain why that gap shrinks as sample size grows.

📄 View solution

Chapter 5 Quick Reference

When σ is unknown and estimated by sample s, use the t-distribution — heavier tails than normal, reflecting the extra uncertainty from estimating s itself
Degrees of freedom (one-sample): df = n − 1 — the exact same denominator as Chapter 9's own Bessel's correction
One-sample t: t = (x̄ − μ₀)/(s/√n) — same shape as the z-statistic, compared against a t-distribution instead
Welch's two-sample t: t = (x̄₁−x̄₂)/√(s₁²/n₁ + s₂²/n₂), with a conservative df = min(n₁,n₂)−1
The t-distribution converges to the normal distribution as n (and therefore df) grows
A visible descriptive difference between two groups doesn't automatically mean a statistically significant one — Chapter 6's own A/B testing chapter builds directly on this
Next chapter: A/B testing in practice