Automate Incident Response with Runbooks
Why and how to automate incident response runbooks: what to automate first, levels of automation, guardrails, and Ansible, AWS SSM and Rundeck examples.
Automating incident response with runbooks turns the steps an on-call engineer would type during an outage into tested code that runs the same way every time. It cuts the minutes lost to gathering diagnostics and applying known fixes, reduces mistakes made under pressure, and leaves an audit trail. Start with diagnostics, then add safe, approved remediation.
This post is about the runbooks themselves: which ones to automate, how to structure them, what guardrails they need, and what they look like in Ansible, AWS Systems Manager and Rundeck. For the alerting and paging side, see our list of tools to automate incident alerts and on-call routing. For running response from chat, see why ChatOps is gaining ground in incident handling.
Key takeaways
- A runbook is a documented procedure for a known operational situation. An automated runbook is the same procedure as executable code with inputs, steps, checks and logs.
- Automate in stages: first diagnostics (read-only), then remediation with a human approval step, then fully automatic remediation for well-understood, low-risk fixes.
- Every automated runbook needs guardrails: least-privilege credentials, input validation, rate limits, a verification step and a clear way to stop it.
- Good candidates are frequent, well understood and low risk: restart a stuck service, clear a full disk, scale out a pool, roll back a failed deploy.
- Restore time is spent error budget, so runbook automation protects SLOs directly.
What a runbook is and why manual ones fall short
A runbook answers the question "an alert just fired; what do I do?" It lists checks to run, likely causes and the fix for each. The Google SRE book introduction reports that recording best practices in a playbook ahead of time speeds up recovery compared with improvising, and the chapter on being on-call notes that stress pushes people toward intuitive rather than deliberate actions.
Manual runbooks, usually a wiki page, have known weaknesses:
- They go stale. Hostnames, commands and dashboards change; the page does not.
- They depend on the reader. Two engineers following the same page at 3 a.m. may do different things.
- They are slow. Copying commands, switching between terminals and pasting output into the incident channel all take minutes.
- They leave a weak record. It is hard to say afterwards exactly what was run, when and by whom.
Automated runbooks fix these because the procedure is code: it is version controlled, reviewed, tested, and it logs every step.
Why automate: the incident timeline view
Break an incident into phases and look at where time goes:
| Phase | Manual work | What automation does |
|---|---|---|
| Detect | Alert fires, engineer is paged | Unchanged; this is monitoring and alerting |
| Acknowledge and gather context | Log in, check dashboards, run status commands | Diagnostic runbook runs on alert and posts results to the incident channel |
| Diagnose | Compare symptoms against known causes | Runbook checks the known causes in order and reports which one matched |
| Remediate | Type commands, often on several hosts | Parameterized remediation with approval, rate limits and verification |
| Verify | Watch graphs until they recover | Runbook checks the health signal and reports success or escalates |
| Review | Reconstruct actions from memory and chat | Execution log is the timeline |
The phases that shrink are context gathering, remediation and verification. These are the parts of time to restore service most under the team's control.
Levels of runbook automation
Teams rarely go from wiki page to self-healing in one step. A staged approach builds trust:
- Level 0: documented. Steps on a page with copy-paste commands.
- Level 1: scripted diagnostics. A read-only job gathers logs, metrics and status and posts them. No risk, immediate time saving.
- Level 2: one-click remediation. The fix is a job the on-call engineer starts with parameters, from a UI, CLI or chat. A human decides; the machine executes.
- Level 3: approved automatic remediation. The alert triggers the runbook, which pauses for approval before changing anything.
- Level 4: self-healing. The runbook runs and remediates without a human for a narrow, well-tested case, and pages only if the fix does not work.
Most runbooks should stop at level 2 or 3. Level 4 suits only fixes that are safe to repeat and whose failure mode you understand completely.
Choosing what to automate first
Look at the last three months of incidents and pages. Score each recurring response on four questions:
| Question | Automate sooner if |
|---|---|
| How often does it happen? | Weekly or more |
| Is the fix always the same? | Yes, with clear conditions |
| What is the worst outcome if the automation misfires? | Minor and reversible |
| How long does the manual version take? | More than a few minutes, or needs several systems |
Typical first candidates:
- Collect diagnostics for a slow service: recent deploys, error logs, pod status, resource usage.
- Restart a service or pod that hits a known memory leak, after capturing a heap dump.
- Clear old logs or temp files when a disk crosses 90%.
- Scale out a worker pool when a queue backs up.
- Roll back the latest deployment when error rate rises right after a release.
- Fail over a read replica or rotate an expired certificate.
This is also how SRE teams reduce toil, described in our guide to SRE practices for DevOps teams.
Anatomy of an automated runbook
- Trigger: manual, an alert webhook, a schedule or a chat command.
- Inputs: typed parameters (instance ID, service name) with validation so a typo cannot target the wrong system.
- Pre-checks: confirm the situation matches the runbook. If the disk is not actually full, stop.
- Action steps: idempotent where possible, so running twice does no harm.
- Approval gate: for anything that changes production state, until the runbook has earned trust.
- Verification: check the health signal that triggered the runbook.
- Escalation: if verification fails, page a human with everything the runbook gathered.
- Logging: every step, input and output to a central log or the incident channel.
Examples in three tools
Ansible playbook: clear space on a full disk
Ansible is a common choice for host-level remediation because playbooks are readable and modules are idempotent. This playbook deletes application logs older than seven days, then checks free space.
- name: Free disk space on app hosts
hosts: "{{ target_hosts }}"
become: true
tasks:
- name: Find app logs older than 7 days
ansible.builtin.find:
paths: /var/log/myapp
patterns: "*.log.*"
age: 7d
register: old_logs
- name: Remove old logs
ansible.builtin.file:
path: "{{ item.path }}"
state: absent
loop: "{{ old_logs.files }}"
- name: Check free space on /var
ansible.builtin.command: df -h /var
register: df_out
changed_when: false
- name: Show result
ansible.builtin.debug:
var: df_out.stdout_lines
Run it with an explicit target, for example ansible-playbook free-disk.yml -e target_hosts=app-prod-07, so it never defaults to every host.
AWS Systems Manager Automation: reboot with approval
Systems Manager Automation runbooks are YAML or JSON documents using schema version 0.3. AWS publishes many predefined runbooks, and you can write your own. The aws:approve action pauses until named principals approve. This custom runbook asks for approval, then reboots an instance:
description: Reboot an EC2 instance after on-call approval
schemaVersion: '0.3'
assumeRole: "{{ AutomationAssumeRole }}"
parameters:
InstanceId:
type: String
allowedPattern: "^i-[a-z0-9]{8,17}$"
AutomationAssumeRole:
type: String
mainSteps:
- name: approve
action: aws:approve
timeoutSeconds: 900
onFailure: Abort
inputs:
Message: "Approve reboot of {{ InstanceId }}"
MinRequiredApprovals: 1
Approvers:
- arn:aws:iam::111122223333:role/OnCallEngineer
- name: reboot
action: aws:executeAwsApi
inputs:
Service: ec2
Api: RebootInstances
InstanceIds:
- "{{ InstanceId }}"
The allowedPattern constraint rejects malformed instance IDs. Automation can be started from EventBridge rules, so a CloudWatch alarm can trigger it. Note from the AWS docs: the Automation free tier is not available to customers new to Automation from 14 August 2025, so check current pricing before running at volume.
Rundeck: one-click jobs for the on-call team
Rundeck (by PagerDuty, with an open source edition and a commercial PagerDuty Runbook Automation product) wraps scripts, Ansible playbooks and API calls in jobs with typed options, access control and execution history. It suits level 2 automation: the on-call engineer picks a job, fills in validated options and runs it, without needing shell access to production.
Kubernetes-native remediation
Kubernetes already automates some runbook steps. Liveness probes restart unhealthy containers, Deployments replace failed pods, and the Horizontal Pod Autoscaler scales on metrics. Write runbooks for what these do not cover, such as a rollback with kubectl rollout undo deployment/checkout after a bad release.
Guardrails that keep automation safe
| Risk | Guardrail |
|---|---|
| Runbook targets the wrong system | Typed, validated inputs; no "all hosts" default |
| Automation loops and makes things worse | Rate limits and a maximum number of runs per hour; stop and page after that |
| Over-broad credentials | A dedicated role with only the permissions the runbook needs |
| Fix hides a real problem | Every automatic remediation opens a ticket, so recurring fixes get a root-cause review |
| Runbook breaks silently after a change | Test runbooks in staging and in game days; keep them in Git with code review |
| Nobody knows it ran | Post start, result and logs to the incident channel |
| Cannot stop a bad run | A kill switch or abort action documented for on-call |
Keeping runbooks current
- Store runbooks in Git next to the service they support, and review changes like application code.
- Link each alert to its runbook in the alert annotation, so the page includes the link or triggers the job.
- After each postmortem, ask whether a runbook should be created, changed or retired.
- Run game days where the team triggers a failure in a test environment and uses the runbook as written.
Postmortems and runbook updates are part of how SRE teams handle major failures; see how SRE teams handle SLA breaches. Faster restore also protects the error budget, explained in why SRE error budgets matter.
Learn the tooling
Writing idempotent Ansible playbooks for operations tasks is covered in our Ansible training in Pune. Monitoring, alerting and incident practice are part of the DevOps training. Both run in the Kharadi classroom and live online. For interview practice, see real-time SRE interview questions on incident drills.
Frequently asked questions
What is an automated runbook?
It is an operational procedure written as executable code, with validated inputs, steps, checks and logging, so a known incident response runs the same way every time.
Why automate incident response with runbooks?
It shortens context gathering and remediation, removes typing mistakes made under pressure, and records exactly what was done, which helps the postmortem.
Should runbooks fix problems without a human?
Only for narrow, well-tested, reversible fixes. Most teams start with automated diagnostics, then one-click remediation, then approval-gated automatic runs.
Which incidents should I automate first?
Frequent ones with a consistent, low-risk fix, such as clearing disk space, restarting a service with a known leak, scaling out workers or rolling back a failed deploy.
What tools are used for runbook automation?
Common choices include Ansible, AWS Systems Manager Automation, Rundeck or PagerDuty Runbook Automation, Azure Automation and Kubernetes controllers, often triggered from alerting or chat tools.
How do I keep automated runbooks from causing outages?
Use validated inputs, least-privilege roles, rate limits, approval gates, a verification step, escalation on failure and a documented way to abort.
How do automated runbooks relate to toil?
Repeated manual incident steps are toil. Automating them frees on-call time for root-cause fixes that stop the incident from recurring.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0