Probability & Statistics Fundamentals' own Chapter 1 opened with a direction: model known, predict the outcome. Its Chapter 9 flipped that briefly — summarize data you already have. This course goes one step further than either: given only a limited, incomplete sample of data, how confidently can you say anything about the full, unobserved reality behind it? That's inference, and it's the genuinely hard, genuinely useful part of applied statistics.

Descriptive vs. Inferential — A Direction Change

Chapter 9's own worked example computed a mean and median from exactly seven sampled response times. It never asked the harder question this course opens with: how much should anyone actually trust a number computed from just seven data points? That's the entire subject of this course.

Five Concrete Connections to Real Work

What This Course Builds Directly On

What This Course Won't Cover

A few genuinely related topics stay deliberately out of scope, to keep this course focused on the core inferential toolkit rather than sprawling into adjacent specialties:

• Advanced experimental design — multi-armed bandits, sequential testing, and other more sophisticated A/B testing variants beyond the fixed-sample-size approach Chapter 6 covers
• Machine learning model evaluation — cross-validation, precision/recall, and related ML-specific metrics belong to Data Science & ML's own ds1/ml1 courses, not here
• Full computational Bayesian methods — MCMC sampling and similar techniques for genuinely complex Bayesian models stay out of scope; Chapter 9 covers Bayesian reasoning, not Bayesian computation at scale

Where This Course Is Headed

Hands-On Exercises

Chapter 1 promised this chapter would build directly on Probability & Statistics Fundamentals' own Central Limit Theorem. Here's the payoff: the CLT isn't just a curiosity about averages — it's the entire mathematical foundation for saying anything trustworthy about a limited sample.

Population vs. Sample

The population is the complete set of everything you'd ideally want to know about — every request that will ever be made, every user who will ever sign up. The sample is the limited subset actually collected and measured. Almost every real measurement in engineering is a sample, not the full population — you never have every request that will ever occur, only the ones logged so far.

Sampling Error — Not a Mistake, Just Randomness

Sampling error is the natural, expected difference between a point estimate and the true population value, arising purely from which subset happened to be sampled — not from anything done wrong. Two different samples of the same size, drawn from the exact same population, will almost always produce two slightly different sample means, purely by chance.

The Sampling Distribution

Imagine repeating the sampling process many times — draw a fresh sample of size n, compute its mean, and repeat. The distribution of all those sample means is the sampling distribution of the sample mean — a genuinely different object from the distribution of individual data points.

Worked Example: Response Times, Revisited

Reusing Probability & Statistics Fundamentals' own response-time model — true population mean μ = 200ms, true population standard deviation σ = 30ms (values a team would rarely know for certain in reality, but useful here to see sampling behavior clearly):

Suppose a team samples n = 25 requests and gets a sample mean of 204ms — 4ms above the true 200ms. Standardizing this as a z-score against the sampling distribution (not individual requests): z = (204 − 200) / 6 = 0.667. That's a thoroughly unremarkable result — well within a single standard error, exactly the kind of harmless wobble sampling error alone would produce. Nothing here suggests anything actually changed.

Sampling Error vs. Sampling Bias — A Critical Distinction

Collecting more data shrinks sampling error, per the σ/√n formula above — but only if the sampling method itself is sound. If the method itself systematically excludes part of the population, more data doesn't fix anything.

Sampling & the Sampling Distribution in Code

Hands-On Exercises

Chapter 2 established that a point estimate always carries sampling error. A confidence interval is the direct, practical fix: instead of reporting a single number as if it were exact, report a range of plausible values, together with how confident that range actually is.

Building a Confidence Interval

The critical value (z*) comes straight from Probability & Statistics Fundamentals' own standard normal distribution — how many standard errors wide the interval needs to be to capture the stated percentage of the sampling distribution:

Worked Example: The Response-Time Sample, Revisited

Reusing Chapter 2's own sample: n = 25 requests, sample mean x̄ = 204ms, σ = 30ms (still assumed known here — Chapter 5's own t-test covers the more realistic case where σ itself must be estimated). Standard error: SE = 30/√25 = 6ms.

Width Shrinks With Sample Size

Reusing Chapter 2's own three sample sizes, at a fixed 95% confidence level:

More data doesn't change the confidence level — it tightens the interval at whatever confidence level was chosen, exactly Chapter 2's own σ/√n shrinkage, now made directly visible in the width of the reported range.

What a Confidence Interval Actually Means

A Forward Note: Proportions

Everything above used a sample mean. Chapter 6's A/B testing material needs confidence intervals for a proportion instead — a conversion rate, not an average response time. The idea is identical; only the standard error formula changes, to SE = √(p(1−p)/n). The confidence interval itself still follows the exact same estimate ± z* × SE shape.

Confidence Intervals in Code

Hands-On Exercises

Chapter 2 ended on an unresolved tension: a z-score of 2.25 was called "notably large," but the chapter admitted it didn't yet have "the formal tool to make that determination with real statistical rigor." This chapter is that tool.

The Null and Alternative Hypothesis

A hypothesis test never proves H₁ — it only asks whether the observed data provides strong enough evidence to reject H₀. Failing to reject H₀ isn't the same as proving it true; it just means the evidence wasn't strong enough to rule it out.

Significance Level — Directly Tied to Chapter 3's Confidence Level

The significance level (α) is the threshold risk of a false rejection a team is willing to accept — conventionally α = 0.05 (5%). It's not a new idea: α = 1 − confidence level, exactly Chapter 3's own 95% confidence level restated from the opposite direction.

The Test Statistic and P-Value

The p-value answers a precise question: if H₀ were actually true, how likely is a result at least this extreme, purely from sampling error? A small p-value means the observed data would be a genuinely unusual coincidence under H₀ — evidence favoring H₁ instead.

Resolving Chapter 2's Own Two Examples

Testing H₀: μ = 200ms against H₁: μ ≠ 200ms, reusing Chapter 2's own two scenarios:

Chapter 2 could only eyeball these results informally. This chapter turns that instinct into a precise, defensible number.

Type I and Type II Errors

Every hypothesis test can go wrong in two distinct ways:

Lowering α reduces Type I errors directly, but — with a fixed sample size — tends to increase Type II errors, since a stricter threshold makes real effects harder to detect too. Neither error can be eliminated without more data.

Two Real Cautions

Hypothesis Testing in Code

Hands-On Exercises

Every z-test in Chapter 4 quietly assumed the population standard deviation, σ, was already known. In real work, that's almost never true — only the sample standard deviation s is available, computed exactly the way Probability & Statistics Fundamentals' own Chapter 9 defined it, with Bessel's n−1 correction. That single substitution changes the math more than it might seem.

Why Estimating σ Changes the Distribution

When σ is estimated from the same limited sample as s, the resulting test statistic carries extra uncertainty on top of ordinary sampling error — and its true sampling distribution is no longer exactly normal. It follows the t-distribution instead: shaped like the normal bell curve, but with heavier tails, reflecting that extra layer of uncertainty from estimating s itself.

The One-Sample t-test

Worked example: a team doesn't know the true σ, only their sample — n = 25, x̄ = 204ms, s = 32ms. Testing H₀: μ = 200:

Same conclusion as Chapter 4's own z-test on similar numbers — unsurprising, since a t-distribution with 24 degrees of freedom is already close to normal.

Comparing Two Groups: The Two-Sample t-test

Comparing two independent groups' means — exactly the shape of an A/B test — uses Welch's t-test, which doesn't assume the two groups share the same variance (a more honest default than assuming equal variances, and the version most statistical software now uses by default):

Worked example: Group A (control), n₁=20, x̄₁=205ms, s₁=25ms; Group B (treatment), n₂=22, x̄₂=190ms, s₂=28ms.

The t-test in Code

Hands-On Exercises

Chapter 5's own visible-but-not-significant response-time gap set up the exact question this chapter answers properly. Real A/B tests almost always compare a proportion — a conversion rate, a click-through rate — not a continuous mean. That needs the proportion-specific version of everything built so far.

The Two-Proportion Z-Test

Comparing two independent conversion rates uses two different standard error formulas, depending on the question:

The distinction matters: the hypothesis test assumes H₀ is true (the two groups share one real rate, so it's honest to pool them into a single best estimate for that shared rate). The confidence interval makes no such assumption — it uses each group's own separate observed rate, since it's estimating how different the two groups genuinely are, not testing whether they're the same.

Worked Example: A Conversion-Rate A/B Test

Control (A): n₁=1000, 80 conversions (p₁=8%). Treatment (B): n₂=1000, 100 conversions (p₂=10%). Validity check: 1000×0.08=80, 1000×0.92=920, both comfortably above 5 for both groups — the normal approximation is safe to use.

The 95% confidence interval for the true difference, using the unpooled SE: 0.02 ± 1.96 × 0.01279 ≈ [−0.005, 0.045] — a range spanning from a small negative effect to a fairly large positive one, consistent with the test's own failure to reach significance. The interval crossing zero and the non-significant p-value are two views of the exact same conclusion.

Sample Size Planning — Was This Test Big Enough?

Statistical power (1 − β, where β is Chapter 4's own Type II error rate) is the probability a test correctly detects a real effect of a given size, if one truly exists. Before running a test, sample size can be planned to hit a target power (conventionally 80%):

For the same 8%→10% effect actually being tested above (p̄ = 0.09):

Why Smaller Effects Need Dramatically More Data

The effect size sits squared in the denominator of the sample size formula — a small change in the effect being detected has an outsized impact on the data required.

Halving the effect size to detect very nearly quadrupled the required sample size — a direct consequence of that squared denominator, and a genuinely important planning reality: reliably detecting small improvements is expensive in data, often far more than intuition suggests.

A Real Trap: Peeking at Results Early

A/B Testing in Code

Hands-On Exercises

Every chapter so far tested a single variable, or compared two groups on one metric. This chapter asks a different question: do two different variables move together? And — the far more important half of the chapter — what an honest "yes" actually does and doesn't let you conclude.

The Pearson Correlation Coefficient

Worked example: CPU usage (%) and response time (ms) across six servers:

r ≈ 0.997 — an extremely strong positive linear relationship: as CPU usage rises, response time rises right along with it, almost perfectly linearly across this data.

The Question This Number Can't Answer

A correlation this strong is tempting to read as "high CPU usage causes slow responses." That might well be true here — but the number r = 0.997 itself doesn't establish it. Any observed correlation between two variables has four possible explanations:

The Classic Confounding Example

A more directly engineering-relevant version: "time spent on a page" correlates positively with "conversion rate" — but this is a strong candidate for reverse causation, not X causing Y. Users who are already leaning toward converting tend to read more carefully and spend longer on the page because they're interested — the extra time is a symptom of interest, not necessarily a cause of the sale.

What Actually Establishes Causation

Correlation in Code

Hands-On Exercises

Linear Algebra Fundamentals' own Chapter 6 planted a forward reference: "fitting a straight line through a set of data points is solving a system of linear equations for the best-fit slope and intercept." This chapter is where that promise is finally kept — and it builds directly on Chapter 7's own correlation work, not separately from it.

Least Squares — Fitting the Best Line

Linear regression fits a line y = mx + b that best predicts y from x. "Best" means minimizing the sum of squared vertical distances between each actual point and the line — least squares.

Worked Example: Fitting a Line to Chapter 7's Own Dataset

Reusing Chapter 7's exact CPU/response-time data:

Residuals — What the Line Doesn't Capture

A residual is actual − predicted for each point — the vertical gap the line missed.

Small, mixed-sign residuals across the board — the line fits this data closely, with no obvious pattern left unexplained.

R² — The Square of Chapter 7's Own r

Chapter 7 already computed r ≈ 0.997 for this exact dataset — no new calculation needed: R² ≈ 0.997² ≈ 0.994. About 99.4% of the variation in response time across these six servers is explained by CPU usage alone in this linear model, leaving only about 0.6% unexplained.

Making a Prediction

Using the fitted line for a CPU value not in the original data, CPU = 65%: response ≈ 2.343 × 65 + 24.21 ≈ 176.5ms.

Linear Regression in Code

Hands-On Exercises

Chapter 1 promised this chapter would generalize Probability & Statistics Fundamentals' own Bayes' Theorem — applied there to a single, fixed piece of evidence — into an ongoing process. Here's the whole idea in one sentence: yesterday's posterior is today's prior.

Two Genuinely Different Philosophies

Every chapter of this course except this one has quietly worked in the frequentist framework: a population parameter (a true mean, a true conversion rate) is a fixed, unknown constant, and probability describes the long-run behavior of the estimation method across repeated sampling — precisely Chapter 3's own correct interpretation of a confidence interval. The Bayesian framework treats the parameter itself as uncertain, with its own probability distribution representing genuine belief — a distribution that gets updated, piece by piece, as evidence arrives.

Building an actual numeric credible interval needs machinery beyond this course's own scope (conjugate priors, the Beta distribution for proportions) — genuinely out of scope here, per Chapter 1's own boundary. This chapter covers Bayesian reasoning, not Bayesian computation at that level.

Sequential Updating: Yesterday's Posterior, Today's Prior

Reusing Probability & Statistics Fundamentals Chapter 4's own spam-filter numbers exactly: P(spam) = 0.40, P("free"|spam) = 0.30, P("free"|not spam) = 0.05. That chapter computed the posterior once: P(spam|"free") = 0.80.

Now a second, independent signal arrives on the same email — a suspicious link, with P(link|spam) = 0.60, P(link|not spam) = 0.02. The trick: the 0.80 posterior from round one becomes the prior for round two.

Two pieces of corroborating evidence, applied one at a time, drove belief from a 40% baseline all the way to over 99% — each round using nothing more than Fundamentals Chapter 4's own Bayes' Theorem, applied again with an updated starting point.

A/B Testing Results, Read the Bayesian Way

Some modern experimentation platforms report results as "94% probability B beats A" rather than Chapter 6's own p-value framing. That's a genuinely Bayesian statement — a direct probability about which variant is actually better — distinct from, and not directly interchangeable with, a frequentist p-value. Both are legitimate, real tools in active use; knowing which framework a specific number came from is what makes it possible to interpret it correctly.

The Honest Tradeoff: Where the Prior Comes From

Sequential Bayesian Updating in Code

Hands-On Exercises

One continuous worked experiment, touching every chapter of this course in the order a real analyst would actually reach for each idea: testing whether a redesigned checkout button genuinely improves conversion, from a historical baseline through a properly powered experiment to a closing Bayesian reframing.

A Full Worked Experiment — Testing a Redesigned Checkout Button

This is, in essence, exactly what a rigorous, real A/B test looks like end to end — every step traceable to a specific chapter of this course, none of it a shortcut around the planning and honesty each one demanded.

What This Course Doesn't Cover

In the interest of an honest accounting: advanced experimental design (multi-armed bandits, sequential testing), machine learning model evaluation, and full computational Bayesian methods were all named in Chapter 1 as deliberately out of scope. This course built the core inferential toolkit those more specialized areas each build on, not a substitute for them.

This Course's Throughline, Restated

Where This Course Connects

Together with its sibling course, this completes the Probability & Statistics arc within Maths for Programmers — Probability & Statistics Fundamentals built the forward-reasoning vocabulary and distributions; this course built the backward-reasoning discipline for drawing honest conclusions from real, limited data. Both connect directly to Technical Support's own diagnostic material (`perfdiag1`/`incident1`) and to Data Science & ML's own `ds1`/`ml1`, where this exact inferential toolkit gets applied at production scale.

Hands-On Exercises