Automate Incident Response with Runbooks

Why and how to automate incident response runbooks: what to automate first, levels of automation, guardrails, and Ansible, AWS SSM and Rundeck examples.

Aug 16, 2025 - 09:58
Nov 10, 2026 - 18:42
 0  63
Automate Incident Response with Runbooks

Automating incident response with runbooks turns the steps an on-call engineer would type during an outage into tested code that runs the same way every time. It cuts the minutes lost to gathering diagnostics and applying known fixes, reduces mistakes made under pressure, and leaves an audit trail. Start with diagnostics, then add safe, approved remediation.

This post is about the runbooks themselves: which ones to automate, how to structure them, what guardrails they need, and what they look like in Ansible, AWS Systems Manager and Rundeck. For the alerting and paging side, see our list of tools to automate incident alerts and on-call routing. For running response from chat, see why ChatOps is gaining ground in incident handling.

Key takeaways

  • A runbook is a documented procedure for a known operational situation. An automated runbook is the same procedure as executable code with inputs, steps, checks and logs.
  • Automate in stages: first diagnostics (read-only), then remediation with a human approval step, then fully automatic remediation for well-understood, low-risk fixes.
  • Every automated runbook needs guardrails: least-privilege credentials, input validation, rate limits, a verification step and a clear way to stop it.
  • Good candidates are frequent, well understood and low risk: restart a stuck service, clear a full disk, scale out a pool, roll back a failed deploy.
  • Restore time is spent error budget, so runbook automation protects SLOs directly.

What a runbook is and why manual ones fall short

A runbook answers the question "an alert just fired; what do I do?" It lists checks to run, likely causes and the fix for each. The Google SRE book introduction reports that recording best practices in a playbook ahead of time speeds up recovery compared with improvising, and the chapter on being on-call notes that stress pushes people toward intuitive rather than deliberate actions.

Manual runbooks, usually a wiki page, have known weaknesses:

  • They go stale. Hostnames, commands and dashboards change; the page does not.
  • They depend on the reader. Two engineers following the same page at 3 a.m. may do different things.
  • They are slow. Copying commands, switching between terminals and pasting output into the incident channel all take minutes.
  • They leave a weak record. It is hard to say afterwards exactly what was run, when and by whom.

Automated runbooks fix these because the procedure is code: it is version controlled, reviewed, tested, and it logs every step.

Why automate: the incident timeline view

Break an incident into phases and look at where time goes:

PhaseManual workWhat automation does
DetectAlert fires, engineer is pagedUnchanged; this is monitoring and alerting
Acknowledge and gather contextLog in, check dashboards, run status commandsDiagnostic runbook runs on alert and posts results to the incident channel
DiagnoseCompare symptoms against known causesRunbook checks the known causes in order and reports which one matched
RemediateType commands, often on several hostsParameterized remediation with approval, rate limits and verification
VerifyWatch graphs until they recoverRunbook checks the health signal and reports success or escalates
ReviewReconstruct actions from memory and chatExecution log is the timeline

The phases that shrink are context gathering, remediation and verification. These are the parts of time to restore service most under the team's control.

Levels of runbook automation

Teams rarely go from wiki page to self-healing in one step. A staged approach builds trust:

  1. Level 0: documented. Steps on a page with copy-paste commands.
  2. Level 1: scripted diagnostics. A read-only job gathers logs, metrics and status and posts them. No risk, immediate time saving.
  3. Level 2: one-click remediation. The fix is a job the on-call engineer starts with parameters, from a UI, CLI or chat. A human decides; the machine executes.
  4. Level 3: approved automatic remediation. The alert triggers the runbook, which pauses for approval before changing anything.
  5. Level 4: self-healing. The runbook runs and remediates without a human for a narrow, well-tested case, and pages only if the fix does not work.

Most runbooks should stop at level 2 or 3. Level 4 suits only fixes that are safe to repeat and whose failure mode you understand completely.

Choosing what to automate first

Look at the last three months of incidents and pages. Score each recurring response on four questions:

QuestionAutomate sooner if
How often does it happen?Weekly or more
Is the fix always the same?Yes, with clear conditions
What is the worst outcome if the automation misfires?Minor and reversible
How long does the manual version take?More than a few minutes, or needs several systems

Typical first candidates:

  • Collect diagnostics for a slow service: recent deploys, error logs, pod status, resource usage.
  • Restart a service or pod that hits a known memory leak, after capturing a heap dump.
  • Clear old logs or temp files when a disk crosses 90%.
  • Scale out a worker pool when a queue backs up.
  • Roll back the latest deployment when error rate rises right after a release.
  • Fail over a read replica or rotate an expired certificate.

This is also how SRE teams reduce toil, described in our guide to SRE practices for DevOps teams.

Anatomy of an automated runbook

  • Trigger: manual, an alert webhook, a schedule or a chat command.
  • Inputs: typed parameters (instance ID, service name) with validation so a typo cannot target the wrong system.
  • Pre-checks: confirm the situation matches the runbook. If the disk is not actually full, stop.
  • Action steps: idempotent where possible, so running twice does no harm.
  • Approval gate: for anything that changes production state, until the runbook has earned trust.
  • Verification: check the health signal that triggered the runbook.
  • Escalation: if verification fails, page a human with everything the runbook gathered.
  • Logging: every step, input and output to a central log or the incident channel.

Examples in three tools

Ansible playbook: clear space on a full disk

Ansible is a common choice for host-level remediation because playbooks are readable and modules are idempotent. This playbook deletes application logs older than seven days, then checks free space.

- name: Free disk space on app hosts
  hosts: "{{ target_hosts }}"
  become: true
  tasks:
    - name: Find app logs older than 7 days
      ansible.builtin.find:
        paths: /var/log/myapp
        patterns: "*.log.*"
        age: 7d
      register: old_logs

    - name: Remove old logs
      ansible.builtin.file:
        path: "{{ item.path }}"
        state: absent
      loop: "{{ old_logs.files }}"

    - name: Check free space on /var
      ansible.builtin.command: df -h /var
      register: df_out
      changed_when: false

    - name: Show result
      ansible.builtin.debug:
        var: df_out.stdout_lines

Run it with an explicit target, for example ansible-playbook free-disk.yml -e target_hosts=app-prod-07, so it never defaults to every host.

AWS Systems Manager Automation: reboot with approval

Systems Manager Automation runbooks are YAML or JSON documents using schema version 0.3. AWS publishes many predefined runbooks, and you can write your own. The aws:approve action pauses until named principals approve. This custom runbook asks for approval, then reboots an instance:

description: Reboot an EC2 instance after on-call approval
schemaVersion: '0.3'
assumeRole: "{{ AutomationAssumeRole }}"
parameters:
  InstanceId:
    type: String
    allowedPattern: "^i-[a-z0-9]{8,17}$"
  AutomationAssumeRole:
    type: String
mainSteps:
  - name: approve
    action: aws:approve
    timeoutSeconds: 900
    onFailure: Abort
    inputs:
      Message: "Approve reboot of {{ InstanceId }}"
      MinRequiredApprovals: 1
      Approvers:
        - arn:aws:iam::111122223333:role/OnCallEngineer
  - name: reboot
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: RebootInstances
      InstanceIds:
        - "{{ InstanceId }}"

The allowedPattern constraint rejects malformed instance IDs. Automation can be started from EventBridge rules, so a CloudWatch alarm can trigger it. Note from the AWS docs: the Automation free tier is not available to customers new to Automation from 14 August 2025, so check current pricing before running at volume.

Rundeck: one-click jobs for the on-call team

Rundeck (by PagerDuty, with an open source edition and a commercial PagerDuty Runbook Automation product) wraps scripts, Ansible playbooks and API calls in jobs with typed options, access control and execution history. It suits level 2 automation: the on-call engineer picks a job, fills in validated options and runs it, without needing shell access to production.

Kubernetes-native remediation

Kubernetes already automates some runbook steps. Liveness probes restart unhealthy containers, Deployments replace failed pods, and the Horizontal Pod Autoscaler scales on metrics. Write runbooks for what these do not cover, such as a rollback with kubectl rollout undo deployment/checkout after a bad release.

Guardrails that keep automation safe

RiskGuardrail
Runbook targets the wrong systemTyped, validated inputs; no "all hosts" default
Automation loops and makes things worseRate limits and a maximum number of runs per hour; stop and page after that
Over-broad credentialsA dedicated role with only the permissions the runbook needs
Fix hides a real problemEvery automatic remediation opens a ticket, so recurring fixes get a root-cause review
Runbook breaks silently after a changeTest runbooks in staging and in game days; keep them in Git with code review
Nobody knows it ranPost start, result and logs to the incident channel
Cannot stop a bad runA kill switch or abort action documented for on-call

Keeping runbooks current

  • Store runbooks in Git next to the service they support, and review changes like application code.
  • Link each alert to its runbook in the alert annotation, so the page includes the link or triggers the job.
  • After each postmortem, ask whether a runbook should be created, changed or retired.
  • Run game days where the team triggers a failure in a test environment and uses the runbook as written.

Postmortems and runbook updates are part of how SRE teams handle major failures; see how SRE teams handle SLA breaches. Faster restore also protects the error budget, explained in why SRE error budgets matter.

Learn the tooling

Writing idempotent Ansible playbooks for operations tasks is covered in our Ansible training in Pune. Monitoring, alerting and incident practice are part of the DevOps training. Both run in the Kharadi classroom and live online. For interview practice, see real-time SRE interview questions on incident drills.

Frequently asked questions

What is an automated runbook?

It is an operational procedure written as executable code, with validated inputs, steps, checks and logging, so a known incident response runs the same way every time.

Why automate incident response with runbooks?

It shortens context gathering and remediation, removes typing mistakes made under pressure, and records exactly what was done, which helps the postmortem.

Should runbooks fix problems without a human?

Only for narrow, well-tested, reversible fixes. Most teams start with automated diagnostics, then one-click remediation, then approval-gated automatic runs.

Which incidents should I automate first?

Frequent ones with a consistent, low-risk fix, such as clearing disk space, restarting a service with a known leak, scaling out workers or rolling back a failed deploy.

What tools are used for runbook automation?

Common choices include Ansible, AWS Systems Manager Automation, Rundeck or PagerDuty Runbook Automation, Azure Automation and Kubernetes controllers, often triggered from alerting or chat tools.

How do I keep automated runbooks from causing outages?

Use validated inputs, least-privilege roles, rate limits, approval gates, a verification step, escalation on failure and a documented way to abort.

How do automated runbooks relate to toil?

Repeated manual incident steps are toil. Automating them frees on-call time for root-cause fixes that stop the incident from recurring.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.