Every course in this subject so far has assumed good documentation already exists somewhere to consult — a ladder to follow, a procedure to reference, a runbook to reach for. This course exists because that assumption is often false, and because writing something down is only the first of two very different achievements. `backup1` spent an entire course establishing that "we have backups" isn't the same claim as "we can recover." This course makes the identical argument about documentation: "we wrote it down" isn't the same claim as "someone under pressure, unfamiliar with the system, can actually follow it and succeed."

What Every Prior Chapter in This Subject Quietly Assumed

Each Technical Support course so far has assumed a piece of documentation exists somewhere, ready to be used. This course exists because writing that documentation, and making it actually work, is a real skill nobody else in this subject has taught:

This course is specifically about producing the thing every other course assumes is already sitting there, ready and correct, when it's actually needed.

Not a Sysadmin-Architecture Course

This course isn't about designing systems or deciding what tooling an organization should use — that's a separate discipline. It's about capturing and communicating knowledge about systems that already exist, so the next person who needs that knowledge doesn't have to rediscover it under pressure.

The Central Claim of This Whole Course

"We wrote it down" confirms only that words exist in a document somewhere. It confirms nothing about whether those words are complete, accurate, still current, or genuinely followable by someone who isn't already the expert who wrote them. Those are different claims, and — exactly as `backup1` argued for backups — only one of them is actually proven by someone genuinely unfamiliar with the system successfully following the document, under real conditions.

What This Course Actually Covers

• Choosing and structuring the right document (Chapters 2–4) — KB article vs. runbook vs. ticket, writing for a reader under pressure, and a runbook's own proper shape
• Capturing and protecting knowledge (Chapters 5–6) — getting tribal knowledge out of people's heads before it's lost, and doing so without creating a security liability
• Keeping documentation actually trustworthy (Chapters 7–8) — fighting silent decay over time, and testing a runbook the way `backup1` tests a backup
• Making it usable (Chapter 9) — organizing documentation so the right person can actually find it when they need it

An Outage That Isn't Resolved Yet

An on-call technician, alone during a real outage at 3am, pulls up an existing runbook for restarting a specific service and follows it step by step. It fails partway through — step 4 references a server hostname that was decommissioned eight months ago during a migration nobody updated the runbook for. The outage takes twice as long to resolve as it should have. Nothing about this is resolved in this chapter — it's deliberately left open. Chapter 8 comes back to it directly, once the material on testing a runbook has actually been covered.

Hands-On Exercises

A support technician produces three genuinely different kinds of documents, and conflating them is one of the most common, most quietly damaging mistakes in this whole subject. Each answers a different question, for a different reader, at a different point in time — knowing which one you're actually writing before you start is the first real skill this course teaches.

Three Documents, Three Questions

Why Conflating Them Actually Hurts

• A runbook padded with background explanation becomes hard to follow step by step under real pressure — the "why" belongs in a linked KB article, not woven into the steps themselves
• A ticket written as if it were a reusable procedure wastes effort generalizing something that may never recur in exactly that form — and if it does recur, it belongs promoted into a real runbook, not left buried in ticket history where nobody will find it again
• A KB article with no procedural content, when the reader actually needed concrete steps, leaves them with theory and no clear action to take

When One Should Become Another

A single ticket documenting one incident is exactly the right document the first time something happens. Once the same problem recurs — the specific pattern `incident1`'s own material teaches recognizing — that recurring pattern deserves promotion into a genuine runbook, not another one-off ticket that repeats the same investigation from scratch. Likewise, if a runbook's own background section keeps growing, that's a sign it should be split out into its own linked KB article, leaving the runbook itself lean and purely procedural.

Comparing the Three Directly

Worked Example: From Ticket to Runbook to KB Article

A specific service crashes under load for the first time — a ticket records what happened and how it was resolved. It happens again, and a third time — the recurring pattern is now promoted into a genuine runbook: the exact steps to mitigate and restart the service safely. Separately, a linked KB article explains why the service crashes under load in the first place — the underlying architectural cause, useful context for understanding the problem, but not itself a set of actionable steps. Three documents, three purposes, each doing its own job.

Hands-On Exercises

Chapter 2 distinguished which document you're writing. This chapter covers how to actually write it — specifically for the reader this whole course keeps returning to: someone stressed, tired, possibly unfamiliar with the system, and quite possibly not the document's own author, trying to resolve a real problem right now.

The Core Design Principle: Assume the Reader Isn't You

Not your knowledge, not your context, not your calm state of mind. Write as if for a stranger under pressure — not a future version of yourself who remembers writing it and already knows what every step means.

Concrete Techniques

• Short, imperative steps — "Run X," not "You might want to consider running X"
• One action per step — never bundle multiple actions into a single numbered instruction
• State the expected result after each step — so the reader can confirm they're actually on track before moving forward, rather than discovering something went wrong three steps later
• No unexplained jargon or internal-only abbreviations — define anything non-obvious on first use, or avoid it entirely
• Concrete over vague — "restart the nginx service," not "restart the web server," if there's any ambiguity about which service that actually means
• Never assume unstated context — "obviously you'd check the logs first" isn't obvious to a stressed stranger; say it explicitly

Why This Matters Specifically at 3am

Stress and fatigue measurably reduce reading comprehension and increase the tendency to skip steps or misread instructions. A document that reads as perfectly clear to a calm, attentive author reviewing their own work may fail completely for a stressed reader encountering it cold. This isn't a claim that Chapter 1's own runbook was badly written in the first place — its failure was a different problem (Chapter 8's own territory) — but it's exactly why writing quality matters more, not less, in documents meant to be used under these specific conditions.

One Clear Path, Fallbacks Kept Separate

Avoid branching, conditional prose in the main procedure — "if this doesn't work, you could try either A or B depending on..." embedded mid-step slows down the common case for every reader, including the vast majority who won't need the fallback at all. State one clear primary path, and keep troubleshooting or fallback steps clearly separated out, not interleaved with the steps most readers will actually follow.

Before and After

Hands-On Exercises

Chapter 3 covered how to write each individual step. This chapter covers the shape the runbook itself needs as a whole — four structural sections that, together, make a runbook genuinely complete and safe to use, not just a list of individually well-written instructions.

The Four Sections

Preconditions Deserve Real Weight

Preconditions are easy to write as a single throwaway line, but skipping them properly is a common way a runbook goes wrong before step 1 even happens. This connects directly to `remote1`'s own material on working safely on a system you don't own — confirming you're actually on the correct system, and that you have the access and authorization the procedure needs, belongs here explicitly, not left as an unstated assumption.

Verification Is Different From a Per-Step Expected Result

Chapter 3's per-step expected results confirm each individual action worked. Verification is a separate, final check confirming the whole procedure achieved its actual goal — every step can complete without error while the underlying problem remains unresolved. A runbook without this final check can report success while leaving the real issue untouched.

Rollback: The Most Neglected Section

People write the happy-path steps carefully and then skip planning for failure — even though a runbook is disproportionately likely to be used exactly when something is already going wrong, which makes hitting a genuine failure partway through more likely, not less, compared to routine documentation used under calm conditions.

Worked Example: A Runbook Skeleton

Hands-On Exercises

Chapters 2 through 4 assumed you already know what needs documenting. This chapter covers a harder problem: knowledge that only exists in one experienced person's head, never written down anywhere at all. This is the more extreme version of Chapter 1's own central claim — here there isn't even a flawed document to fall back on, only what one person happens to remember.

Why This Is a Real, Common Risk

An experienced technician often "just knows" how to handle a specific recurring quirk without ever having written it down — it works fine as long as they're around, and becomes a serious gap the moment they're not, whether that's a resignation, a transfer, or simply being unreachable during an incident that happens to hit while they're on leave.

Why Simply Asking Isn't Enough

People are notoriously bad at fully verbalizing procedural knowledge they perform automatically — steps that have become second nature get silently skipped when someone's asked to just describe what they do, not because they're hiding anything, but because they genuinely stop consciously noticing those steps.

Watch, Don't Just Ask

The single most effective technique is watching someone actually perform the task and writing down what they actually do — not what they said they'd do beforehand. The gap between the interview answer and the observed reality is often exactly where the missing documentation was hiding all along.

Two Questions That Draw Out the Rest

• "What would you do if X specific thing went wrong?" — asked for several plausible failure variations, this draws out troubleshooting and rollback knowledge (Chapter 4's own territory), which tends to be the most tribal knowledge of all, since the failure paths are rehearsed far less often than the happy path
• "How would you know if this actually worked?" — draws out verification knowledge (also Chapter 4's own territory) that experienced people often check unconsciously without realizing they're doing it at all

Timing: This Is Proactive Work, Not an Emergency

Waiting until someone announces they're leaving is often too late to do this thoroughly — a rushed knowledge-transfer exercise days before someone's last day rarely reproduces the depth a calm, ongoing capture process would. This works best as a routine part of normal operations, not a crisis response.

Whose Knowledge to Prioritize

Start with the people who've been in a specific role or system the longest, and specifically anyone who's the only person who knows how to do a particular task — a genuine single point of failure in the team's own knowledge, worth identifying and addressing deliberately rather than discovering by accident when they're unavailable.

From Notes to a Real Document

Once captured, interview and observation notes are raw material, not a finished product — they still need to become an actual runbook or KB article, following Chapter 2's own document-type distinction and Chapters 3–4's own writing and structural principles, not left sitting as unstructured notes nobody else can easily use.

Hands-On Exercises

Chapters 2 through 5 pushed toward documentation that's more complete, more detailed, and captures more of what an experienced person actually knows. That same completeness has a real cost: the more thoroughly useful a document is to a legitimate technician resolving an incident quickly, the more useful it can potentially be to someone who shouldn't have it at all. This chapter is about writing genuinely useful documentation without it becoming an attack tool sitting in plain sight.

The Core Rule: Never Embed Real Sensitive Values

Never write an actual password, a real API key, or a genuine sensitive configuration value directly into a runbook or KB article — even an internal one, even one you trust everyone currently reading it. Use a clear placeholder instead (<YOUR_API_KEY>), and reference a proper secrets vault or access-management process rather than the literal value.

Why "It's Internal Only" Isn't a Sufficient Excuse

This is `secsupport1`'s own least-exposure principle, applied specifically to documentation content itself, not just live data handling during a ticket. Internal documentation systems get compromised too — an insider misusing legitimate access (`secsupport1`'s own curiosity-access material), or a single compromised account with ordinary read access to the internal wiki. Documentation systems are frequently subject to less scrutiny than production systems themselves — the same reasoning `backup1` applied to backup infrastructure being an attractive, under-monitored target applies directly here.

Beyond Credentials: Attack-Blueprint-Level Detail

Some non-credential detail is still risky to document in full — exact firewall rule logic, or specific detection thresholds precise enough to let someone deliberately stay just under an alert threshold. A runbook can be specific enough to be genuinely useful to a legitimate reader without functioning as a literal blueprint for anyone else who happens to get hold of it.

Access Control on the Documentation Itself

Not every runbook needs to be visible to the entire organization. Documentation covering security-incident handling or disaster-recovery procedures deserves the same least-exposure access thinking `secsupport1` and `backup1` already applied to systems and backups — restrict access to whoever genuinely needs it, rather than defaulting to "everyone can see everything" purely because that's the more convenient default.

Reference, Don't Reveal

Both right-hand versions give a legitimate reader exactly what they need to proceed, without the document itself ever holding the actual sensitive value.

Hands-On Exercises

Chapters 2 through 6 covered producing genuinely good, secure documentation. This chapter covers what happens to it afterward — because a document doesn't stay accurate forever, and the way it goes wrong is exactly the same shape `backup1` described for silent backup failures: nothing announces that a document has become wrong. It sits there, looking complete and authoritative, until someone follows it and it fails — precisely what happened in Chapter 1's own outage scenario.

The Direct Parallel to backup1

A document that was accurate when written is exactly like a backup job that completed successfully when it ran — both can silently stop being trustworthy as the underlying reality changes, with no alert, no error, and no visible signal that anything is now wrong.

What Causes Decay

• Systems change — servers get renamed or decommissioned (exactly Chapter 1's own scenario), software gets upgraded, procedures get updated — without the documentation being updated to match
• Organizational change — the person who wrote it leaves, and Chapter 5's own captured tribal knowledge can go stale too if nobody maintains it after the original author is gone
• Nobody owns it — no single person or team is actually responsible for keeping a specific document accurate, so it's genuinely nobody's job to notice when it's wrong

Why Decay Is Genuinely Silent

Unlike a broken link or an obvious formatting error, a runbook with an outdated hostname looks completely normal — it reads clearly, has all four sections Chapter 4 described, and follows every writing principle from Chapter 3. The only way to actually discover it's wrong is to try using it live (which is exactly the discovery method Chapter 1's own outage suffered through) or to deliberately audit it against current reality before that happens — Chapter 8's own testing practice.

Combating Decay

• Explicit ownership — every significant document has a named owner responsible for it, not left as an ownerless artifact nobody feels accountable for
• A defined review cadence — mirroring `backup1`'s own testing cadence, a periodic scheduled review rather than "whenever someone happens to notice"
• Tying documentation updates into the change process itself — when a system changes, updating anything that references it should be part of that change's own checklist, not a separate afterthought easy to forget
• A visible "last reviewed" date — an honest, immediate signal letting a reader judge how stale a document might be, even before discovering whether it's actually wrong

Hands-On Exercises

Chapter 7 explained why documentation decays silently. This chapter delivers the actual practice that closes the gap — the exact discipline `backup1`'s own Chapter 5 built for backups, applied here to documentation: only someone genuinely attempting to follow a document proves it actually works.

Why the Author Testing Their Own Document Isn't Enough

The author already knows what they meant — which means they'll unconsciously fill in every gap and resolve every ambiguity a genuine stranger would trip over. This is the same underlying problem Chapter 5's own "watch, don't just ask" material identified: an expert's own automatic knowledge quietly contaminates their ability to notice what's actually missing from what they wrote down.

What Counts as a Genuine Test

• Ideal: someone unfamiliar with the specific procedure actually executes it, in a safe test environment when the real action would be risky or destructive
• A reasonable substitute: when live execution genuinely isn't safe or practical, a careful, skeptical walkthrough — checking that every referenced hostname, value, and tool actually still exists and is current — mirroring `backup1`'s own "sampling beats nothing" reasoning for backup sets too large to fully restore-test
• What to confirm: that preconditions are genuinely checkable as written, steps produce the stated expected results, verification actually confirms the real goal, and rollback is itself followable — Chapter 4's own four sections, each independently confirmed to hold up

Test Cadence, Tied to Chapter 7's Own Review Material

A test is a stronger, more rigorous form of the review Chapter 7 already established, not a separate unrelated task. Critical runbooks — the ones reached for during high-severity incidents — deserve more frequent, more rigorous testing than a rarely-used KB article ever needs.

Documenting Test Results

Each test should record the date, who tested it, what was found, and whether anything needed fixing — creating a real trail of evidence, echoing `log1`'s own foundational discipline, rather than an untracked, fading sense that a document "still seems fine."

Hands-On Exercises

Chapters 1 through 8 covered producing documentation that's accurate, secure, and genuinely tested. This chapter covers the last practical problem: a technically excellent document nobody can actually find in the moment they need it fails its own purpose just as completely as a wrong one would — just through a different mechanism entirely.

Findability Is a Separate Problem From Quality

A perfectly accurate, well-tested runbook buried in the wrong folder, named inconsistently, or scattered across three slightly different copies fails a stressed reader just as thoroughly as an inaccurate one would. All the work from every prior chapter in this course is wasted if the reader simply can't locate the document at the moment it matters.

Consistent Naming and Location

A predictable naming and location scheme lets a reader guess where something likely lives, or search effectively, rather than needing to already know a specific document exists before they can find it. Consistency matters more than any particular convention — the value comes from a reader being able to predict the pattern, not from the pattern itself being perfect.

Tagging: Multiple Paths to the Same Document

Tag by system, by severity or criticality, and by document type (Chapter 2's own KB article/runbook/ticket distinction) — a reader searching under pressure might know the system name but not the specific problem, or the reverse, so more than one path to the same document genuinely helps.

Cross-Linking Related Documents

A runbook should link directly to its own supporting KB article — Chapter 2's own "split a growing background section into its own linked article" pattern — and related runbooks for the same system should reference each other, so a reader who finds one has a clear path to the others they might also need.

The Duplicate-Copy Problem: A Sneaky Variant of Decay

Chapter 7 covered a document going stale where it lives. This is a related but distinct problem: once a document is updated, old copies pasted elsewhere — a chat message, a different wiki page, a printed page left on a desk — don't get updated along with it, and can be found and followed instead of the current version. The original document can be perfectly current while an outdated copy of it, still findable somewhere else, actively misleads whoever happens to find that copy first.

Deprecate, Don't Silently Delete or Silently Leave in Place

When a document is genuinely retired — the procedure no longer applies, the system it describes was decommissioned — mark it clearly as deprecated or archived. Silently deleting it loses potentially useful historical context; silently leaving it in place looking current means someone can find and follow it by accident, the same duplicate-copy problem above, but applied to the original document itself once it's no longer accurate.

Hands-On Exercises

Nine chapters built the toolkit — document types, writing for a stressed reader, structure, tribal-knowledge capture, security-conscious writing, decay, testing, and findability. This capstone runs three fresh tasks through that toolkit end to end, matching the three-scenario shape most of this subject's own courses use.

Task 1: Promoting a Recurring Ticket Into a Real Runbook

A specific API integration has failed intermittently three times over two months, each time handled as its own separate ticket.

Task 2: Capturing a Departing Senior Technician's Knowledge

A senior technician who's the only person who really understands a quirky legacy subsystem is leaving in a few weeks.

Task 3: Auditing Runbooks Before a Disaster-Recovery Review

A scheduled review of the organization's critical runbooks, ahead of an upcoming disaster-recovery exercise.

Chapter Attribution

Honest Scope Note

Hands-On Exercises