Bayesian Inference & Updating Beliefs

Statistical Inference & Applied Statistics

Chapter 9 · Bayesian Inference & Updating Beliefs

Chapter 1 promised this chapter would generalize Probability & Statistics Fundamentals' own Bayes' Theorem — applied there to a single, fixed piece of evidence — into an ongoing process. Here's the whole idea in one sentence: yesterday's posterior is today's prior.

Two Genuinely Different Philosophies

Every chapter of this course except this one has quietly worked in the frequentist framework: a population parameter (a true mean, a true conversion rate) is a fixed, unknown constant, and probability describes the long-run behavior of the estimation method across repeated sampling — precisely Chapter 3's own correct interpretation of a confidence interval. The Bayesian framework treats the parameter itself as uncertain, with its own probability distribution representing genuine belief — a distribution that gets updated, piece by piece, as evidence arrives.

Chapter 3's own warning, finally resolved

Chapter 3 warned that a 95% confidence interval does not mean "95% probability the true value is in this interval" — that intuitive-sounding claim is actually the correct definition of a Bayesian credible interval instead, a genuinely different object built from a posterior distribution rather than repeated sampling. The interpretation everyone instinctively reaches for isn't wrong about statistics in general — it's just describing the other framework.

Building an actual numeric credible interval needs machinery beyond this course's own scope (conjugate priors, the Beta distribution for proportions) — genuinely out of scope here, per Chapter 1's own boundary. This chapter covers Bayesian reasoning, not Bayesian computation at that level.

Sequential Updating: Yesterday's Posterior, Today's Prior

Reusing Probability & Statistics Fundamentals Chapter 4's own spam-filter numbers exactly: P(spam) = 0.40, P("free"|spam) = 0.30, P("free"|not spam) = 0.05. That chapter computed the posterior once: P(spam|"free") = 0.80.

Now a second, independent signal arrives on the same email — a suspicious link, with P(link|spam) = 0.60, P(link|not spam) = 0.02. The trick: the 0.80 posterior from round one becomes the prior for round two.

Round	Prior going in	Evidence	Posterior
1	0.40	Contains "free"	0.80
2	0.80 (round 1's posterior)	Contains a suspicious link	≈ 0.992

Two pieces of corroborating evidence, applied one at a time, drove belief from a 40% baseline all the way to over 99% — each round using nothing more than Fundamentals Chapter 4's own Bayes' Theorem, applied again with an updated starting point.

This is exactly how a naive Bayes spam classifier works

Real spam filters combine many word-level and metadata signals this same way — updating a probability sequentially (or equivalently, all at once assuming the evidence pieces are conditionally independent given the hypothesis, the "naive" assumption naive Bayes is named for). Data Science & ML's own ml1 covers this as an actual classification algorithm; here it's the same underlying math, without the ML framing.

A/B Testing Results, Read the Bayesian Way

Some modern experimentation platforms report results as "94% probability B beats A" rather than Chapter 6's own p-value framing. That's a genuinely Bayesian statement — a direct probability about which variant is actually better — distinct from, and not directly interchangeable with, a frequentist p-value. Both are legitimate, real tools in active use; knowing which framework a specific number came from is what makes it possible to interpret it correctly.

The Honest Tradeoff: Where the Prior Comes From

A Bayesian analysis depends on a choice, not just the data

Every Bayesian update starts from a prior — and that prior has to come from somewhere: past data, domain expertise, or a deliberately "uninformative" default. Two reasonable analysts can pick two different reasonable priors and land on genuinely different posteriors from the identical evidence. This is a real, honest tradeoff, not a flaw to be embarrassed about — frequentist methods avoid needing a prior at all, at the cost of the less intuitive interpretation Chapter 3 already covered. Neither framework is simply "better"; they trade one kind of honesty for another.

Sequential Bayesian Updating in Code

def bayes_update(prior, p_evidence_given_h, p_evidence_given_not_h):
    p_not_h = 1 - prior
    p_evidence = (p_evidence_given_h * prior) + (p_evidence_given_not_h * p_not_h)
    return (p_evidence_given_h * prior) / p_evidence

# Round 1: reusing Fundamentals Ch.4's own spam filter numbers
posterior_1 = bayes_update(prior=0.40, p_evidence_given_h=0.30, p_evidence_given_not_h=0.05)
print(posterior_1)  # 0.8

# Round 2: the round-1 posterior becomes the round-2 prior
posterior_2 = bayes_update(prior=posterior_1, p_evidence_given_h=0.60, p_evidence_given_not_h=0.02)
print(posterior_2)  # 0.9917...

Hands-On Exercises

Exercise 1

Reusing Probability & Statistics Fundamentals Chapter 4's own fraud-detection example (P(fraud) = 0.001, P(flagged|fraud) = 0.95, P(flagged|not fraud) = 0.02, giving a posterior of P(fraud|flagged) ≈ 0.0454), a second signal now fires on the same transaction: an unusual login location, with P(location|fraud) = 0.70, P(location|not fraud) = 0.10. Using this chapter's own sequential-updating method, compute the new posterior after both pieces of evidence.

📄 View solution

Exercise 2

A colleague says "a 90% confidence interval and a 90% credible interval mean the same thing, just from two different calculation methods." Using this chapter's own frequentist-vs-Bayesian distinction, explain specifically what's wrong with this claim.

📄 View solution

Exercise 3

Two analysts investigate the same new fraud-detection signal. Analyst A, who has seen many false alarms from similar signals before, starts with a skeptical prior. Analyst B, newer to the team, starts with a more neutral prior. Given the exact same evidence, explain why they could reasonably reach different posteriors, and why this doesn't mean one of them made a mathematical error.

📄 View solution

Chapter 9 Quick Reference

Frequentist: parameters are fixed constants; probability describes the estimation method's long-run behavior (Chapter 3's own CI interpretation)
Bayesian: parameters have their own probability distribution, representing genuine belief, updated as evidence arrives
A 95% credible interval genuinely does mean "95% probability the true value is here" — the intuitive interpretation Chapter 3 warned doesn't apply to a frequentist CI
Sequential updating: yesterday's posterior becomes today's prior — repeated application of Fundamentals Chapter 4's own Bayes' Theorem
This same mechanism, applied to many signals, is the basis of a naive Bayes classifier — real ML tooling, same underlying math
A Bayesian analysis depends on the prior chosen — a real, honest tradeoff against frequentist methods' own less intuitive interpretation, not a flaw
Next chapter: Capstone — designing and analyzing a real experiment