Structuring a Runbook: Preconditions, Steps, Verification & Rollback

Documentation & Runbooks

Chapter 4 · Structuring a Runbook: Preconditions, Steps, Verification & Rollback

Chapter 3 covered how to write each individual step. This chapter covers the shape the runbook itself needs as a whole — four structural sections that, together, make a runbook genuinely complete and safe to use, not just a list of individually well-written instructions.

The Four Sections

1. Preconditions

What must be true before starting — required access, tools, and confirming you're actually on the correct system.

2. Steps

The numbered procedure itself, following Chapter 3's own writing principles — one action and an expected result per step.

3. Verification

An explicit, end-to-end check confirming the actual goal was achieved — not just that the last step ran without error.

4. Rollback

What to do if something goes wrong partway through, or the procedure completes without actually fixing the problem.

Preconditions Deserve Real Weight

Preconditions are easy to write as a single throwaway line, but skipping them properly is a common way a runbook goes wrong before step 1 even happens. This connects directly to `remote1`'s own material on working safely on a system you don't own — confirming you're actually on the correct system, and that you have the access and authorization the procedure needs, belongs here explicitly, not left as an unstated assumption.

Verification Is Different From a Per-Step Expected Result

Chapter 3's per-step expected results confirm each individual action worked. Verification is a separate, final check confirming the whole procedure achieved its actual goal — every step can complete without error while the underlying problem remains unresolved. A runbook without this final check can report success while leaving the real issue untouched.

Rollback: The Most Neglected Section

People write the happy-path steps carefully and then skip planning for failure — even though a runbook is disproportionately likely to be used exactly when something is already going wrong, which makes hitting a genuine failure partway through more likely, not less, compared to routine documentation used under calm conditions.

No rollback section is an unstated assumption of its own
A runbook without a rollback section implicitly assumes the procedure will always succeed — exactly the kind of assumption Chapter 1's own outage scenario demonstrates can't be trusted. Every runbook needs an explicit answer to "what do I do if this goes wrong partway through," not silence on the question.

Worked Example: A Runbook Skeleton

RUNBOOK: Restarting a Stuck Background Worker PRECONDITIONS - Confirm you are connected to the correct production host (check hostname matches the current, current-as-of-today worker server — see Chapter 1) - Confirm you have sudo access to the worker service account STEPS 1. Run: systemctl status worker.service Expected: shows "active (running)" but with a queue depth above 500 2. Run: systemctl restart worker.service Expected: no error output 3. Run: systemctl status worker.service Expected: shows "active (running)" with queue depth actively decreasing VERIFICATION - Confirm queue depth has dropped below 100 within 5 minutes of restart - Confirm no new errors appear in worker.log during that window ROLLBACK - If queue depth does not drop after 10 minutes, stop the worker service and escalate per incident1's own escalation process — do not attempt a second restart without investigating further

Hands-On Exercises

Exercise 1

Explain why verification is described as genuinely different from a per-step expected result, and give an example of a runbook that could pass every per-step check while still failing verification.

📄 View solution
Exercise 2

Explain why rollback is described as more likely to be needed in a runbook than in routine documentation, using this chapter's own reasoning about when runbooks actually get used.

📄 View solution
Exercise 3

Explain why confirming preconditions is connected directly to `remote1`'s own material on working safely on a system you don't own, rather than treated as a separate, unrelated concern.

📄 View solution

Chapter 4 Quick Reference

  • Preconditions: required access, tools, and confirming you're on the correct system — not a throwaway line
  • Steps: the numbered procedure, per Chapter 3's own writing principles
  • Verification: an end-to-end check confirming the actual goal was achieved, separate from per-step results
  • Rollback: what to do if something goes wrong — the most commonly neglected section, despite being disproportionately needed
  • A runbook missing rollback implicitly assumes the procedure always succeeds
  • Next: Chapter 5, capturing tribal knowledge before it walks out the door