Bayes' Theorem

Probability & Statistics Fundamentals

Chapter 4 · Bayes' Theorem

Chapter 3 ended with a promise: a way to solve for P(B|A) when only P(A|B) is actually known. That tool is Bayes' Theorem — and it delivers one of the most consistently surprising, genuinely useful results in this entire course.

Deriving Bayes' Theorem

Chapter 3's general multiplication rule gave two equal expressions for the same joint probability:

Starting point (Chapter 3)

P(A ∩ B) = P(A|B) × P(B) = P(B|A) × P(A)

Dropping the middle term and dividing both sides by P(A) isolates P(B|A) directly:

Bayes' Theorem

P(B|A) = [P(A|B) × P(B)] / P(A)

Written in the notation it's usually taught with — H for hypothesis, E for evidence:

Bayes' Theorem, standard form

P(H|E) = [P(E|H) × P(H)] / P(E)

Each term has a name worth knowing: P(H) is the prior (what you believed before seeing the evidence), P(E|H) is the likelihood, P(E) is the evidence's overall probability, and P(H|E) is the posterior — the updated belief after accounting for the evidence.

The Law of Total Probability — Filling in the Denominator

P(E) is rarely given directly — it needs to be built from the two ways evidence can occur: when the hypothesis is true, and when it's false.

Law of total probability

P(E) = P(E|H) × P(H) + P(E|not H) × P(not H)

The Classic Example: A Diagnostic Test

A test for a rare disease is genuinely quite accurate:

Quantity	Value
P(D) — disease prevalence	1% (a rare disease)
P(+\|D) — sensitivity	99% — correctly flags 99% of people who have it
P(+\|not D) — false positive rate	5% — incorrectly flags 5% of healthy people

Question: given a positive result, what's the actual probability of having the disease?

Step	Calculation	Result
P(+), total probability	0.99×0.01 + 0.05×0.99	0.0594
P(D\|+), Bayes' Theorem	(0.99×0.01) / 0.0594	0.1667 (≈16.7%)

A 99%-accurate test, and only a 16.7% chance of actually having the disease

This is the correct answer, not a mistake. Because the disease is rare, the sheer number of healthy people who get a false positive (5% of the huge "healthy" group) swamps the smaller number of genuinely sick people who correctly test positive (99% of the tiny "sick" group). This is sometimes called base-rate neglect — ignoring how rare something actually is leads to badly overestimating how meaningful a positive result really is.

A Second Worked Example: A Spam Filter

Reusing this course's own forward reference from Chapter 1 — a spam filter reasoning about a single word:

Quantity	Value
P(spam) — prior	40% of all email is spam
P("free"\|spam)	30% of spam emails contain the word "free"
P("free"\|not spam)	5% of legitimate emails contain "free"

Step	Calculation	Result
P("free"), total probability	0.30×0.40 + 0.05×0.60	0.15
P(spam\|"free"), Bayes' Theorem	(0.30×0.40) / 0.15	0.80 (80%)

Seeing "free" in an email raises the belief it's spam from a 40% prior all the way to an 80% posterior — a real, quantified update, exactly the mechanism named in Chapter 1's own connections table.

Why This Matters for Real Alerting Systems

The same math applies directly to anomaly/intrusion detection

An intrusion-detection system with a 99% true-positive rate and a 1% false-positive rate sounds excellent. But if genuinely malicious connections are extremely rare (say, 1 in 10,000), the exact same base-rate-neglect math applies: an alert firing is correct only a small fraction of the time — the overwhelming majority of alerts are false positives, purely because malicious events are so rare relative to the flood of ordinary traffic. This is the precise mathematical root of alert fatigue, a genuine operational problem this subject's sibling Technical Support courses (secsupport1, incident1) deal with directly from the response-process side; this chapter explains exactly why it happens numerically.

Bayes' Theorem in Code

def bayes(p_evidence_given_h, p_h, p_evidence_given_not_h):
    p_not_h = 1 - p_h
    p_evidence = (p_evidence_given_h * p_h) + (p_evidence_given_not_h * p_not_h)
    return (p_evidence_given_h * p_h) / p_evidence

# The diagnostic test example
p_disease_given_positive = bayes(p_evidence_given_h=0.99, p_h=0.01, p_evidence_given_not_h=0.05)
print(p_disease_given_positive)  # 0.16666... — only ~16.7%

# The spam filter example
p_spam_given_free = bayes(p_evidence_given_h=0.30, p_h=0.40, p_evidence_given_not_h=0.05)
print(p_spam_given_free)  # 0.8 — 80%

Hands-On Exercises

Exercise 1

A fraud-detection system flags 95% of genuinely fraudulent transactions (P(flagged|fraud) = 0.95) and incorrectly flags 2% of legitimate transactions (P(flagged|not fraud) = 0.02). Only 0.1% of all transactions are actually fraudulent (P(fraud) = 0.001). Using Bayes' Theorem, compute P(fraud|flagged) — the actual probability a flagged transaction is really fraud.

📄 View solution

Exercise 2

A different spam filter: 30% of email is spam (P(spam) = 0.30), 25% of spam emails contain the word "winner" (P("winner"|spam) = 0.25), and 1% of legitimate emails contain "winner" (P("winner"|not spam) = 0.01). Compute P(spam|"winner") using this chapter's own Bayes' Theorem method.

📄 View solution

Exercise 3

An intrusion-detection system has a 99% true-positive rate (P(alert|malicious) = 0.99) and a 1% false-positive rate (P(alert|not malicious) = 0.01). Only 1 in 10,000 connections on the network is actually malicious. Compute P(malicious|alert), and explain — using this chapter's own base-rate-neglect finding — what this result means for whether the system is genuinely "good enough" to alert a human on every trigger.

📄 View solution

Chapter 4 Quick Reference

Bayes' Theorem: P(H|E) = [P(E|H) × P(H)] / P(E) — derived directly from Chapter 3's general multiplication rule
Prior P(H), likelihood P(E|H), posterior P(H|E) — the belief before and after accounting for evidence
Law of total probability: P(E) = P(E|H)P(H) + P(E|not H)P(not H) — needed whenever P(E) isn't given directly
A highly accurate test can still produce a low posterior when the underlying condition (the prior) is rare — base-rate neglect
The same math explains why real intrusion/anomaly-detection alerts are often mostly false positives, and why alert fatigue is a real, quantifiable consequence, not just an inconvenience
Next chapter: Random variables and expected value