What is a runbook and how should it be linked to monitoring alerts?

A runbook (also called a playbook) is a documented set of procedures that an on-call engineer follows when a specific alert fires. A well-written runbook dramatically reduces MTTR by pre-answering the first questions an engineer asks: What is this alert? Why does it matter? What checks do I run first? What are the common causes? What do I do if cause A? What do I do if cause B? Who do I escalate to?

Runbooks should be linked directly from the alert definition so they are one click away at 3 AM when the engineer is half-awake. In Alertmanager, you can add an annotations.runbook_url label to each alert rule. Datadog, PagerDuty, and most alerting platforms support custom annotation fields for this purpose.

A good runbook contains:

• Alert summary: What fired and what service it covers.
• Severity and SLO impact: Is the error budget burning? How fast?
• Diagnosis steps: Specific PromQL or log queries to run, with expected outputs for common failure modes.
• Mitigation options: Rollback command, feature flag to disable, cache flush procedure.
• Escalation path: Who to call if the runbook does not resolve the issue within N minutes.
• Post-incident: Link to the postmortem template.

Runbooks become stale quickly. Pair them with a last-updated timestamp and require post-incident updates when the actual resolution differed from documented steps.