Runbooks: Writing Operational Procedures That Work

Runbooks

A runbook is a procedure written for someone who was asleep four minutes ago, is looking at a system they may not have built, and has to make it work now. That reader has no working memory for context, no patience for prose, and no ability to infer what you meant. Everything about how a runbook is written follows from imagining that person.

The consequence is a document that looks nothing like normal technical writing. No narrative, no background, no explanation of why the architecture is the way it is. Instead: a symptom you can match against, a check you can run, a command you can copy, and an expected output you can compare. Numbered steps, literal values, explicit escalation.

The other property that distinguishes a good runbook is that it's linked from the alert that fires. A runbook nobody can find during an incident is a document that exists for audit purposes only.

TL;DR

Write for a tired, unfamiliar reader under pressure. No prose, no context, no inference required.
Every alert links to its runbook. An alert without one is an incomplete alert.
Commands must be copy-pasteable and literal — no placeholders the reader has to resolve under stress.
Include expected output so the reader can tell whether a step worked.
Structure: symptoms → impact → diagnosis → mitigation → verification → escalation.
Mitigate first, diagnose later. Restoring service outranks understanding the cause.
State explicit escalation criteria — who, when, and how, with names or rotations.
A runbook that never changes is stale; a runbook executed identically every time should be automated.

Quick Example

Two properties of that document are what make it work at 3 a.m.: every command can be copied without editing, and every step tells you where to go next based on what you observed.

Core Concepts

Structure

The ordering is deliberate: mitigation appears before deep diagnosis because restoring service is the priority. A runbook that walks the reader through root-cause analysis before offering a rollback is optimized for the wrong outcome.

Runbook vs. playbook vs. postmortem

An incident response playbook covers who declares an incident, who's the incident commander, how status updates are communicated, and when to involve legal or PR. Runbooks are the technical procedures it invokes.

Not just incidents

Routine operational procedures deserve runbooks too, and these are often higher-value because they're executed more often:

Rotating a credential or certificate
Restoring a database backup — and this one should be tested, not just written
Scaling a service for an expected traffic event
Onboarding or offboarding infrastructure access
Failover and failback drills
Data migration and backfill procedures

A backup restore procedure that has never been executed is a hypothesis, not a runbook. The number of organizations that discover their backups don't restore during the incident is not small.

Linking from alerts

The runbook_url annotation flows through to PagerDuty and Slack, so the responder gets the procedure in the notification rather than having to search for it. Make it a hard requirement: an alert without a runbook link doesn't ship. This also has a useful side effect — it forces the question "what should someone actually do about this?", and alerts with no good answer usually shouldn't be paging anyone.

Decision trees

Real diagnosis branches. Make the branches explicit:

A tired reader following a linear list of twelve checks will lose their place. A tree tells them which branch they're on and what to skip.

Best Practices

Write commands that can be copied without editing

Where a value genuinely varies, show how to obtain it in the preceding step rather than leaving a placeholder.

Include expected output

Without it, the reader doesn't know whether the step succeeded. This is the single most common omission and the one that most often causes a responder to proceed on a false assumption.

Put mitigation before root cause

The instinct to understand before acting is correct in normal engineering and wrong during an outage. Structure the document so the fastest path to service restoration is the first thing the reader reaches, with investigation steps clearly marked as post-mitigation.

Add a "last verified" date and honor it

A runbook referencing a decommissioned service or an old command is worse than none, because it consumes the responder's scarcest resource. Review runbooks quarterly, or after any incident where one was wrong, and record the date visibly.

Fix the runbook during the incident

The best moment to correct a runbook is while you're discovering it's wrong. Open a PR with the correction as part of the incident follow-up, not as a backlog item nobody picks up.

Include what not to do

Warnings about tempting-but-wrong actions are frequently the highest-value content in a runbook, because they encode a mistake someone already made.

Test them in game days

Run a scheduled exercise where someone unfamiliar with the system follows the runbook to resolve a simulated failure. Every gap surfaces immediately, and the cost is a scheduled afternoon rather than an incident. See Chaos Engineering.

Automate the ones that never branch

A runbook whose steps are executed identically every time is a script waiting to be written. Automate it, and keep a runbook describing how to run the automation, what it does, and what to do when it fails.

Common Mistakes

Prose instead of steps

Placeholders in commands

No escalation criteria

"Escalate" without a name, a threshold, and a channel means the responder hesitates, and hesitation during an outage is expensive.

An alert with no runbook

Untested procedures

Diagnosis before mitigation

FAQ

Where should runbooks live?

Wherever they're fastest to reach during an incident, which means indexed, searchable, and linked from alerts. In-repo Markdown published to a docs site is the docs-as-code answer and works well. A wiki is acceptable if search is good. The one hard requirement: they must be available when your infrastructure is down — a runbook hosted only on the cluster that's on fire is not a runbook.

How detailed should they be?

Detailed enough that someone who didn't build the system can follow them. That's the bar, and it's higher than most authors assume — you know what "check the queue depth" means and the reader at 3 a.m. may not. The test is to hand it to someone from another team and watch them try.

Who writes them?

The team that owns the service, ideally the person who built the thing or who responded to the incident that motivated it. A runbook written by someone who wasn't there is missing the details that matter. Review by someone unfamiliar is how you find the assumed knowledge.

How do we keep them current?

Three mechanisms that actually work: fix the runbook during the incident where it was wrong, review runbooks as part of every postmortem, and run periodic game days that execute them. Scheduled quarterly reviews help but are the weakest of the three, because reading a runbook rarely reveals what's wrong with it.

Should we automate runbooks away?

Yes, where the procedure is deterministic. A sequence of commands executed identically every time should be a script, an operator, or an automated remediation. Keep a runbook for the automation — what it does, how to run it manually, how to tell if it worked, and what to do when it fails. Fully automated remediation still needs a human-readable description of what just happened.

What about AI-assisted incident response?

Increasingly useful for the correlation and summarization layers — surfacing related past incidents, summarizing a noisy log stream, or drafting the incident timeline. Less reliable for deciding what to do, since it will confidently propose actions with real consequences. The practical pattern today is AI suggesting the likely runbook and summarizing signals, with a human executing the mitigation steps.