Sampling & the Sampling Distribution

Statistical Inference & Applied Statistics

Chapter 2 · Sampling & the Sampling Distribution

Chapter 1 promised this chapter would build directly on Probability & Statistics Fundamentals' own Central Limit Theorem. Here's the payoff: the CLT isn't just a curiosity about averages — it's the entire mathematical foundation for saying anything trustworthy about a limited sample.

Population vs. Sample

The population is the complete set of everything you'd ideally want to know about — every request that will ever be made, every user who will ever sign up. The sample is the limited subset actually collected and measured. Almost every real measurement in engineering is a sample, not the full population — you never have every request that will ever occur, only the ones logged so far.

Point estimate

A point estimate is a single number, computed from a sample, used to estimate an unknown population parameter — a sample mean x̄ estimating the true population mean μ, for instance.

Sampling Error — Not a Mistake, Just Randomness

Sampling error is the natural, expected difference between a point estimate and the true population value, arising purely from which subset happened to be sampled — not from anything done wrong. Two different samples of the same size, drawn from the exact same population, will almost always produce two slightly different sample means, purely by chance.

The Sampling Distribution

Imagine repeating the sampling process many times — draw a fresh sample of size n, compute its mean, and repeat. The distribution of all those sample means is the sampling distribution of the sample mean — a genuinely different object from the distribution of individual data points.

This is exactly Probability & Statistics Fundamentals Chapter 8's own CLT

That course's own Central Limit Theorem already proved this sampling distribution approaches Normal(μ, σ²/n) as n grows — regardless of the shape of the original population. Its own standard error formula, SE = σ/√n, is the standard deviation of this sampling distribution. Nothing new needs deriving here — this chapter is that formula, put to direct use.

Worked Example: Response Times, Revisited

Reusing Probability & Statistics Fundamentals' own response-time model — true population mean μ = 200ms, true population standard deviation σ = 30ms (values a team would rarely know for certain in reality, but useful here to see sampling behavior clearly):

Sample size n	Standard error (σ/√n)
25	6.0ms
100	3.0ms
900	1.0ms

Suppose a team samples n = 25 requests and gets a sample mean of 204ms — 4ms above the true 200ms. Standardizing this as a z-score against the sampling distribution (not individual requests): z = (204 − 200) / 6 = 0.667. That's a thoroughly unremarkable result — well within a single standard error, exactly the kind of harmless wobble sampling error alone would produce. Nothing here suggests anything actually changed.

Sampling Error vs. Sampling Bias — A Critical Distinction

Collecting more data shrinks sampling error, per the σ/√n formula above — but only if the sampling method itself is sound. If the method itself systematically excludes part of the population, more data doesn't fix anything.

A real trap: sampling bias survives any amount of extra data

Suppose a team only measures response times during daytime traffic, missing a slower nightly batch job that runs every night. No matter how many daytime samples they collect — a thousand, a million — their estimate of "true average response time across all traffic" stays systematically wrong, consistently too fast. More data makes a biased estimate more precisely wrong, not more accurate. Sampling error is fixed by collecting more data; sampling bias is fixed only by fixing the sampling method itself.

Sampling & the Sampling Distribution in Code

import math, random

def standard_error(sigma, n):
    return sigma / math.sqrt(n)

print(standard_error(30, 25))   # 6.0
print(standard_error(30, 100))  # 3.0

# A quick simulation: draw many samples, see the sample-mean spread shrink
def simulate_sample_means(mu, sigma, n, num_samples=1000):
    return [
        sum(random.gauss(mu, sigma) for _ in range(n)) / n
        for _ in range(num_samples)
    ]

means_n25 = simulate_sample_means(200, 30, n=25)
means_n100 = simulate_sample_means(200, 30, n=100)
# The spread of means_n100 will be visibly tighter than means_n25 —
# run it and check with statistics.stdev() against the SE formula above

Hands-On Exercises

Exercise 1

A metric's true population standard deviation is σ = 200 users. A sample of n = 25 days gives a sample mean 90 users above the assumed population mean. Compute the standard error, then the z-score for this sample mean against the sampling distribution. Is this result unremarkable (well within 1–2 standard errors) or notably large?

📄 View solution

Exercise 2

A metric has population standard deviation σ = 80. Compute the standard error for sample sizes n = 4, n = 16, and n = 64. Describe the pattern in how much n must grow to halve the standard error, connecting it back to Probability & Statistics Fundamentals' own Chapter 8 finding.

📄 View solution

Exercise 3

A company estimates customer satisfaction using only responses from an optional online contact form — a small, self-selected fraction of all customers. Their estimated satisfaction score is very high. Using this chapter's own sampling-error-vs-sampling-bias distinction, explain why collecting 10 times more responses through that exact same method would not fix the underlying problem with this estimate.

📄 View solution

Chapter 2 Quick Reference

Population = everything you'd ideally want to know; sample = the limited subset actually collected
Point estimate: a single sample statistic (e.g., x̄) used to estimate an unknown population parameter (e.g., μ)
Sampling error is the natural, expected sample-to-sample variation — not a mistake
Sampling distribution of the sample mean: approaches Normal(μ, σ²/n) per the CLT — SE = σ/√n is its standard deviation
Sampling error shrinks with more data; sampling bias (a systematically unrepresentative method) does not — more biased data is just more precisely wrong
Next chapter: Confidence intervals