Data Collection Methods

Research Methodology

Chapter 8 · Data Collection Methods

Every data collection method distorts what it measures in some real, characteristic way. This chapter covers four common methods and two real, well-documented cases — one showing how people misreport their own behavior, and one showing how a famous "textbook fact" about observation itself turned out to be far shakier than its own reputation.

Surveys

Scalable, structured — but relies entirely on honest, accurate self-report

Interviews

Genuine depth and follow-up — but small samples and real interviewer-effect risk

Experiments

Strongest real causal claims — but real observation itself can change behavior

Observational Studies

Real, naturalistic behavior — but no controlled comparison, and real observer bias risk

Surveys: A Real Social-Desirability Case

A Real, Direct Headcount vs. Self-Report

Sociologists C. Kirk Hadaway, Penny Long Marler, and Mark Chaves, in a 1993 American Sociological Review study, physically located and counted attendance at every church in a rural Ohio county, comparing it directly against self-reported church attendance from public opinion surveys. The real headcount came out to roughly half the attendance rate people reported on surveys.

The authors attributed most of the gap to social desirability bias — people overreporting behavior they believe is socially valued. Later researchers debated exactly how much of the gap comes from overreporting versus incomplete headcounts and other measurement issues — a real, honest reminder that even a well-designed comparison study can leave its own precise mechanism genuinely disputed, even when the headline gap itself is well established.

Experiments and the Real Hawthorne Effect Story

Between 1924 and 1932, researchers at Western Electric's Hawthorne Works — later reinterpreted by a Harvard team led by Elton Mayo — varied factory lighting and other conditions, and reported that worker productivity rose regardless of the change, supposedly because workers were changing their behavior simply from knowing they were being observed. That story became the textbook "Hawthorne effect."

A Real, Documented Reanalysis

Economists Steven Levitt and John List (2011), building on an earlier reanalysis by Jones (1992), went back to the actual original data and found the dramatic "output rose no matter what changed" pattern largely collapses once real day-of-week patterns and pay-period incentives are properly controlled for. The Hawthorne effect isn't fully debunked — but the version repeated in most textbooks is a genuinely weaker piece of evidence than its own decades-long reputation suggests.

Four Methods, Compared

Method	Real strength	Real weakness
Survey	Scales to large samples cheaply	Relies on accurate self-report — real social desirability bias risk
Interview	Real depth, follow-up questions possible	Small samples, real interviewer-effect risk
Experiment	Strongest support for real causal claims	Observation itself can change the behavior being measured
Observational study	Captures real, naturalistic behavior	No controlled comparison; real observer bias risk

The Real, Practical Lesson

Every method distorts its own data in a specific, predictable direction — surveys toward social desirability, experiments toward observation effects. Choosing a method well means anticipating its own characteristic distortion in advance, not discovering it after the data's already collected.

Hands-On Exercises

Exercise 1

A company wants to know how often employees actually take real lunch breaks (versus working through them). Explain why a self-report survey on this specific topic carries a real social-desirability risk, using the church attendance case as a model, and propose one alternative or supplementary method that would reduce that risk.

📄 View solution

Exercise 2

Explain, in your own words, what specifically the Levitt and List reanalysis changed about the real, appropriate lesson to draw from the Hawthorne studies — not "the Hawthorne effect never happened," but something more precise.

📄 View solution

Exercise 3

You need to know both "what percentage of customers are satisfied" and "what specifically frustrates the dissatisfied ones." Choose one method from this chapter for each sub-question, and explain why a single method couldn't answer both well.

📄 View solution

Chapter 8 Quick Reference

Surveys, interviews, experiments, and observational studies each distort data in their own characteristic way
Hadaway, Marler & Chaves' real 1993 study found self-reported church attendance roughly double the real, physically counted rate
The exact cause of that gap (overreporting vs. measurement issues) remains genuinely debated, even though the headline gap itself is well established
The classic 1924-1932 Hawthorne studies' "observation alone boosts output" story is real but, per Levitt & List (2011) and Jones (1992), weaker evidence than its textbook reputation suggests
Choosing a data collection method well means anticipating its own predictable distortion in advance