Exercise 1: Why the Author Reviewing Their Own Runbook Tests the Wrong Thing — Possible Solution ==================================================================== WHAT CHAPTER 5's "WATCH, DON'T JUST ASK" MATERIAL ESTABLISHED ------------------------------ Per Chapter 5, interviews alone are unreliable because "people are notoriously bad at fully verbalizing procedural knowledge they perform automatically - steps that have become second nature get silently skipped." An expert's own automatic knowledge causes them to omit details from a description of their own process, not because they're hiding anything, but because those details no longer register as distinct, conscious actions to them. WHY THE SAME PROBLEM APPLIES TO AN AUTHOR REVIEWING THEIR OWN WRITTEN DOCUMENT ------------------------------ Per this chapter, "the author already knows what they meant - which means they'll unconsciously fill in every gap and resolve every ambiguity a genuine stranger would trip over." Reading their own runbook, the author's own background knowledge silently supplies exactly the missing context an unfamiliar reader wouldn't have - they read an ambiguous step and correctly infer what it means, without noticing that the ambiguity exists at all, because their own understanding papers over it automatically. WHY THIS MEANS THE REVIEW CAN'T ACTUALLY FIND THE GAPS ------------------------------ The entire purpose of testing a runbook is finding places where a reader without the author's own background knowledge would get stuck or make a wrong assumption. An author reviewing their own document is structurally incapable of experiencing that gap, because they bring exactly the knowledge the test is supposed to check for the absence of. The review can confirm the document reads correctly to someone who already understands the system - which is the one thing that was never actually in question. WHY "TESTS THE WRONG THING" IS THE RIGHT DESCRIPTION, NOT "IS INSUFFICIENT" ------------------------------ This isn't a matter of the author's review being a weaker or partial version of a real test - it's checking an entirely different property (does this make sense to someone who already knows the system) than the one that actually matters (does this make sense to someone who doesn't). Per this chapter's own warn-box, "confirming a document with its own author ... tests the wrong thing entirely." WHY THIS WORKS AS AN ANSWER ------------------------------ It restates Chapter 5's own reasoning about automatic knowledge causing omissions, explains why the identical mechanism applies to an author reviewing their own writing, and explains why this means the author's review checks an entirely different property than the one a genuine test actually needs to verify.