Writing a runbook somebody will actually read

Most runbooks are read exactly twice: once by the person reviewing the handover, and once at 02:40 by someone who is not the author, does not have context, and needs the answer in under two minutes.

The second reader is the one the document is for. Almost every convention in a bad runbook comes from writing for the first.

What a runbook is not

• Not an architecture document. A service diagram is useful and belongs in a different file. Nobody debugging a stalled queue at 3am needs a paragraph on why the broker was chosen.
• Not a design rationale. Reasoning matters during review and is noise during an incident.
• Not an API reference. Link to it. A copy goes stale, and a stale runbook is worse than none, because it costs time before it costs nothing.
• Not a document that assumes access. “Restart the worker” is useless to somebody who cannot log in. Getting access is part of the procedure.

A runbook is a set of answers to one question: this specific thing is wrong, what do I do?

Organise by failure mode, not by component

The most common structural mistake is a table of contents that mirrors the system: one section per service, each describing what it does.

That ordering requires the reader to already know which component is at fault. If they knew that, they would be halfway to the fix. The reader has a symptom, not a diagnosis.

Organise by symptom instead. The headings should read like the things people actually say:

• Orders are not appearing in the warehouse system
• The nightly report did not arrive
• Customers report 401 errors on the public API
• The retry queue is growing
• Everything is slow

Under each symptom, in this order: how to confirm it is really this, the most likely causes ranked by frequency, the fix for each, and how to verify you fixed it. A reader should be able to scan the headings, find their symptom, and be executing a command within thirty seconds.

What every entry needs

For each failure mode:

1. Symptom. What the reporter sees, in their words, including the literal error text, because that is what people paste into search.
2. Confirm. One command or dashboard link that proves this is the right section. It prevents the expensive mistake of applying the wrong fix confidently.
3. Likely causes, ordered by how often they happen. Not alphabetically, not by severity. Frequency.
4. Fix. Exact copy-pasteable commands with placeholders marked. Not “restart the service” but the literal invocation.
5. Verify. How to know it worked, from the outside. A green log line is not verification. The order appearing in the warehouse system is.
6. If that did not work. Who to contact, with a name and a channel, and what to record before escalating.

Above the failure modes, a short block that rarely changes: where the system runs and where the logs are, as links; which credentials it uses and where they live; who owns it; what is safe to restart and what is not, with the reason; and what breaks if it is down for an hour. That fits on one screen and answers most of what a newcomer needs.

Write commands, not descriptions

The difference between a runbook that works and one that does not is usually specificity. Weak: check whether the sync worker is processing messages. Strong:

# 1. Confirm the queue is draining. depth should fall over 30s.
queuectl depth --queue orders-sync
sleep 30
queuectl depth --queue orders-sync

# 2. If depth is flat and greater than 0, check the consumer.
kubectl -n prod get pods -l app=orders-sync
kubectl -n prod logs -l app=orders-sync --tail=100 --since=15m

The second version can be executed by someone who has never seen the system. The first version requires them to already know how. Every place you write a description instead of a command, you have moved work from the calm present to the stressed future.

Mark anything destructive clearly and say what it costs. “This drops in-flight messages, up to about 200 at peak hours” is the kind of sentence that stops a bad decision.

Test it by handing it to somebody else

An untested runbook is a draft. The test is simple and slightly uncomfortable. Pick someone who did not build the system, give them the runbook and the access it lists, break something in staging, and then sit behind them and say nothing.

Write down every one of these:

• Where they pause. A pause means a step assumed knowledge they do not have.
• What they ask you. Every question is a missing line.
• Where they open a different tab. That is a link the document should have contained.
• What they get wrong. Ambiguity, not incompetence.

Do not explain. The instinct to explain is what produced the gaps, because explaining patches the reader instead of the document. Repeat once or twice a year, and after any significant change. A runbook decays at the speed of the system it describes.

A worked example

Here is one entry, complete, from a workflow that pushes invoices to an accounting system.

Invoices stopped appearing in the accounting system

Symptom. Finance reports missing invoices for a given day. The operations channel may show fetched: 0 from the nightly status post.

Confirm. Open the run history for invoice-push and check the last successful run timestamp. If it is older than 26 hours, this is the right section.

Cause 1, most common: the API token expired. Tokens last 90 days. Fix: generate a new token under Settings, API access, then update the accounting_api credential in the automation platform. Verify: run invoice-push manually for yesterday and confirm one invoice arrives.

Cause 2: the workflow was left deactivated, usually after debugging. Fix: reactivate it. Verify: the next scheduled run writes a row to job_status.

Cause 3: the billing API is returning empty pages. Check its status page. If it is up, compare the request date range against the timezone setting, which has caused this twice.

If none apply. Paste the last failing run URL and the response body into the operations channel and tag the owner. Do not replay more than once, since replays are not deduplicated.

Three causes, in the order they actually occur, each with a fix and a verification. Short is the point. A person can read that block in ninety seconds and be doing something useful.

The test that matters

Before you call a runbook finished, ask: if the person who built this system were unreachable for two weeks, could somebody else keep it running using only this document and the access it lists?

If the answer is no, the runbook is not done. If the answer is yes, it is worth more than the code it describes, because the code can be read and the operational knowledge cannot.