Literature Reviews: Synthesizing Without Cherry-Picking

Research Methodology

Chapter 5 · Literature Reviews: Synthesizing Without Cherry-Picking

A literature review's whole job is to summarize what a body of evidence actually shows — not what a writer already believed before reading it. This chapter covers the real structural difference between a loosely organized review and a rigorously protocoled one, a well-documented reason evidence itself skews toward positive findings, and a real, severe case of what happens when a review leaves out the studies that don't fit.

Narrative vs. Systematic Review

Narrative review	Systematic review
Sources chosen at the author's own discretion	Sources found via an explicit, pre-defined, reproducible search protocol
No fixed inclusion/exclusion criteria	Inclusion/exclusion criteria set and published in advance
Genuinely useful for broad context and framing	Genuinely stronger for answering one specific, well-defined question
Real risk: selection bias toward the author's own expectations	Real strength: another researcher can, in principle, reproduce the same search

PRISMA: A Real, Structured Protocol

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, first published in 2009 and substantially updated in 2020, gives systematic reviewers a real, standardized checklist and a flow diagram tracking every record from initial search through screening, eligibility review, and final inclusion — making the whole selection process auditable rather than hidden inside a single author's own judgment.

The File Drawer Problem

A Real, Documented Bias in the Evidence Itself

Psychologist Robert Rosenthal's 1979 paper "The file drawer problem and tolerance for null results," published in Psychological Bulletin, named a real, structural bias: studies finding a significant effect are far more likely to be submitted and accepted for publication than studies finding no effect. Rosenthal's own description: journals fill up with the studies that found something, while the file drawers fill up with the ones that didn't.

This means even a perfectly honest, exhaustive review of published literature can still be systematically skewed — not because anyone cherry-picked, but because the underlying pool of published studies was already tilted before the review ever began.

A Real Case Where Selective Reporting Caused Harm

In 2001, "Study 329" — a clinical trial of the antidepressant paroxetine in adolescents, funded by SmithKline Beecham (now GlaxoSmithKline) — was published reporting the drug was both effective and safe for treating adolescent depression.

The Real 2015 Reanalysis Reversed Both Findings

In 2015, the RIAT (Restoring Invisible and Abandoned Trials) initiative — formed specifically to address selective reporting in clinical trials — published a full reanalysis of the original trial data in BMJ. It found paroxetine was no more effective than placebo, the opposite of the original paper's own claim — and that the paroxetine group had more than twice as many severe adverse events and four times as many psychiatric adverse events, including suicidal behaviors and self-harm, compared to placebo.

The Real, Practical Lesson

A literature review built only on the original 2001 publication would have synthesized confidently wrong conclusions on both efficacy and safety — not from any error in the review's own writing, but because the underlying source it relied on had itself selectively reported its results. Honest synthesis means actively seeking out reanalyses, raw data, and disconfirming studies, not just the version of a finding that was published first or most widely cited.

Hands-On Exercises

Exercise 1

Explain, in your own words, why a systematic review's own explicit, published inclusion/exclusion criteria make it more resistant to cherry-picking than a narrative review — even though both review types could technically cover the exact same set of studies.

📄 View solution

Exercise 2

Explain, in your own words, why the file drawer problem can bias a literature review's conclusions even if the reviewer never intentionally cherry-picked anything and read every single published study they could find.

📄 View solution

Exercise 3

In the Study 329 case, the 2001 and 2015 papers analyzed the exact same underlying trial data, yet reached opposite conclusions. Explain what this tells you about the real difference between "a study was conducted" and "a study's own published conclusions can be fully trusted."

📄 View solution

Chapter 5 Quick Reference

Narrative reviews use author discretion; systematic reviews follow explicit, published inclusion/exclusion criteria
PRISMA (2009, updated 2020) gives systematic reviewers a real, standardized checklist and flow diagram for transparency
Rosenthal's real 1979 "file drawer problem" describes how published literature skews toward positive results, even without deliberate cherry-picking
GSK's Study 329 (2001) claimed paroxetine was effective and safe for adolescents; a real 2015 RIAT reanalysis of the same data found neither was true
Honest synthesis requires actively seeking reanalyses and disconfirming evidence, not just the first or most-cited published version of a finding