Statistical Reasoning Traps: Base Rates & the Prosecutor's Fallacy

Critical Thinking

Chapter 5 · Statistical Reasoning Traps: Base Rates & the Prosecutor's Fallacy

Ignoring how rare something actually is in the first place is a real, well-documented reasoning error — one that even trained doctors get wrong at a striking rate, and one that has genuinely sent an innocent person to prison. This chapter covers both, with real numbers behind each.

Base Rate Neglect: A Real Study, Real Doctors, Real Wrong Answers

Casscells, Schoenberger & Grayboys, 1978, New England Journal of Medicine

Researchers asked 60 people — house officers, fourth-year medical students, and attending physicians at Harvard Medical School teaching hospitals — a real question: if a disease affects 1 in 1,000 people, and a test for it has a 5% false positive rate, what's the real chance a person who tests positive actually has the disease? The most common answer given was 95%. The average answer was 56%. Only 18% of respondents gave the real correct answer: roughly 2%.

The real math: out of 1,000 people, 1 actually has the disease. Of the remaining 999 healthy people, 5% (about 50 people) will still test positive — a false positive. So out of roughly 51 total positive results, only 1 is a true case. That's the base rate — how rare the disease is to begin with — genuinely swamping the test's own apparent accuracy.

The Prosecutor's Fallacy: A Real, Tragic Legal Case

In 1999, English solicitor Sally Clark was convicted of murdering her two infant sons, who had each died in infancy months apart. Pediatrician Roy Meadow testified that the odds of two children in an affluent family both dying of Sudden Infant Death Syndrome (SIDS) naturally were 1 in 73 million — arrived at by squaring his own estimated 1-in-8,500 probability for a single SIDS death.

The Real, Documented Statistical Error

Meadow's figure was flawed twice over. First, squaring the individual probability wrongly assumed the two deaths were statistically independent, ignoring that shared genetic or environmental factors within one family make a second SIDS death more likely once a first has occurred. Second, and more fundamentally, it committed the prosecutor's fallacy — treating "the probability of two natural deaths, if she's innocent" as though it were the same number as "the probability she's innocent, given two deaths." Those are genuinely different questions, and conflating them is a real, common, and serious statistical error. The Royal Statistical Society later issued a public statement condemning the "misuse of statistics in the courts."

Clark's conviction was overturned on appeal in January 2003, after further evidence came to light. She had already served over three years in prison for a crime the underlying statistics never actually supported.

The Same Underlying Mistake

Case	What was ignored
Harvard doctors	How rare the disease actually is (1 in 1,000) before trusting a positive test
Sally Clark	How rare a wrongful conviction versus a genuine coincidence actually is, once the fallacy is corrected

The Real, Practical Lesson

Both cases share the same underlying error: treating "how unlikely is this evidence, assuming innocence or health" as though it directly answers "how likely is guilt or disease, given this evidence." They're genuinely different questions, and the real base rate — how rare the underlying condition or event actually is — has to be factored in before the second question can be answered honestly.

Hands-On Exercises

Exercise 1

Redo the chapter's own real math with a different, rarer disease: 1 in 10,000 people, with the same 5% false positive rate. Out of 10,000 people, how many total positive results would there be, and what real percentage of those positives would actually have the disease?

📄 View solution

Exercise 2

Explain, in your own words, the specific difference between "the probability of this evidence, assuming innocence" and "the probability of innocence, given this evidence" — using Meadow's real 1-in-73-million figure as the example.

📄 View solution

Exercise 3

Explain, in your own words, why assuming the two deaths in the Clark case were statistically independent was a real, separate error from the prosecutor's fallacy itself — what would need to be true for squaring the individual probability to have been valid?

📄 View solution

Chapter 5 Quick Reference

A real 1978 Harvard Medical School study found most doctors badly overestimated a positive test's real meaning — the correct answer (~2%) was given by only 18% of respondents
The real math: out of 1,000 people with a 1-in-1,000 disease and a 5% false positive rate, only 1 of ~51 total positives is a true case
Sally Clark's real 1999 wrongful murder conviction relied on pediatrician Roy Meadow's flawed 1-in-73-million statistic
The figure wrongly assumed independence between the two deaths and committed the real prosecutor's fallacy
The Royal Statistical Society publicly condemned the misuse; Clark's conviction was overturned in 2003