10 SRE Automation Tools That Cut Toil

10 SRE automation tools mapped to the toil they remove: Terraform, Ansible, Argo CD, Alertmanager, Sloth, Rundeck, KEDA, cert-manager, Litmus, k6.

Dec 17, 2025 - 12:38
Nov 6, 2026 - 08:35
 0  200
10 SRE Automation Tools That Cut Toil

Ten automation tools that remove the most toil for SRE teams are Terraform, Ansible, Argo CD, Prometheus Alertmanager, Sloth, Rundeck (PagerDuty Runbook Automation), KEDA, cert-manager, LitmusChaos and Grafana k6. Each one replaces a repetitive manual job: provisioning, fixing drift, deploying, routing alerts, writing SLO alerts, running runbooks, scaling, renewing certificates, testing failure and testing load.

This post is organised by the toil each tool removes, with a small working example for each. It is not a general catalogue of SRE software. For the broader list of monitoring, incident and observability platforms, see our companion post on the top 10 SRE tools. For what the SRE role covers, start with the role of SREs in DevOps teams. Tool features were checked against official project documentation in November 2026.

Key takeaways

  • Google's SRE book defines toil as manual, repetitive, automatable operational work with no lasting value that grows with the service. Automate the toil you can measure first.
  • Keep a toil log for two weeks: what you did by hand, how often, and how long it took. It tells you which tool to adopt.
  • Declarative tools (Terraform, Argo CD, cert-manager, KEDA) remove whole classes of toil because a controller keeps reconciling to the desired state.
  • Generate SLO alerts from a spec with Sloth instead of hand-writing burn-rate rules for each service.
  • Automate runbooks step by step: document, script, add a button, then trigger from alerts once the steps are proven safe.
  • Every automation needs logs, limits and an off switch. Unbounded automation can turn a small incident into a large one.

Toil, and how to choose what to automate

The Google SRE book describes toil as work tied to running a production service that is manual, repetitive, automatable, tactical, without enduring value, and that scales linearly as the service grows. Its guidance is to keep an SRE's operational work, toil included, under half of their time so the rest goes to engineering that reduces future toil.

A practical way to choose: score each manual task by frequency multiplied by time spent, then weight up anything that is risky when done by hand (production changes at 3 a.m.) or that delays other teams (access requests, environment creation). The highest scores are your first automation targets. The table below maps common toil to the tool in this list that addresses it.

Recurring manual taskToolAutomation style
Creating and changing cloud infrastructure by consoleTerraform / OpenTofuDeclarative plan and apply from Git
Fixing configuration drift on serversAnsibleIdempotent playbooks, scheduled or triggered
Running deploy scripts and fixing cluster driftArgo CDGitOps reconciliation
Sorting duplicate alerts during an incidentPrometheus AlertmanagerGrouping, inhibition, routing
Writing burn-rate alert rules per serviceSlothGenerate rules from an SLO spec
Running the same diagnostic and restart stepsRundeck / PagerDuty Runbook AutomationSelf-service and alert-triggered jobs
Adjusting replicas for queue backlogsKEDAEvent-driven autoscaling
Renewing TLS certificatescert-managerAutomatic issuance and renewal
Manual failover drillsLitmusChaosScheduled chaos experiments
Ad hoc load tests before launchesGrafana k6Load tests with pass/fail thresholds in CI

Infrastructure and configuration toil

1. Terraform (and OpenTofu)

Terraform describes infrastructure in HCL, shows a plan of changes, and applies them through provider APIs. For SREs the toil reduction is consistency: environments are created the same way every time, changes are reviewed in pull requests, and terraform plan run on a schedule reveals drift made by hand in a console. HashiCorp licenses Terraform under the Business Source License; OpenTofu is the open-source fork with the same workflow.

# nightly drift check in CI: exit code 2 means changes are pending
terraform init -input=false
terraform plan -detailed-exitcode -input=false -lock=false
# 0 = no changes, 1 = error, 2 = drift or pending changes

2. Ansible

Ansible applies idempotent playbooks over SSH with no agent. SRE teams use it for OS patching in batches, rotating configuration, restarting services in a controlled order, and repairing drift found by monitoring. The serial keyword limits how many hosts change at once, and max_fail_percentage stops a rollout that is going wrong.

- name: Rolling patch of web tier
  hosts: web
  become: true
  serial: "20%"
  max_fail_percentage: 10
  tasks:
    - name: Apply security updates
      ansible.builtin.dnf:
        name: "*"
        security: true
        state: latest
    - name: Reboot if required
      ansible.builtin.reboot:
        reboot_timeout: 600

3. Argo CD

Argo CD watches Git and keeps Kubernetes clusters in the declared state. Deploys stop being scripts someone runs; they are merges. With automated sync and self-heal enabled, a manual change in the cluster is reverted to what Git says, which removes a whole category of "who changed this?" investigations. Rollback is a Git revert.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/apps
    path: checkout/overlays/prod
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Read more on how GitOps works.

Alerting and SLO toil

4. Prometheus Alertmanager

Alertmanager receives alerts from Prometheus and decides what reaches a human. Grouping turns fifty pod alerts into one notification. Inhibition suppresses symptom alerts while a cause alert is firing, for example service alerts while the whole cluster is down. Silences mute known work during maintenance. Routing sends each alert to the right team and channel. All of this cuts the manual triage that wears out on-call engineers.

route:
  receiver: default
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
inhibit_rules:
  - source_matchers: ['alertname="ClusterDown"']
    target_matchers: ['severity="warning"']
    equal: ['cluster']
receivers:
  - name: default
  - name: oncall-pager

Pair it with Grafana dashboards as described in using Prometheus and Grafana together.

5. Sloth

Writing multi-window, multi-burn-rate alerts by hand for each service is slow and error prone. Sloth takes a short SLO specification and generates the Prometheus recording rules and alerting rules for you, following the burn-rate approach from the Google SRE Workbook. It can run as a CLI in CI or as a Kubernetes controller.

version: "prometheus/v1"
service: "checkout"
slos:
  - name: "requests-availability"
    objective: 99.9
    description: "Checkout API non-5xx responses"
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="checkout",code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
    alerting:
      name: CheckoutHighErrorBudgetBurn
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

Pyrra is a similar open-source project if you prefer a UI. Our post on SLOs and business goals explains how to choose the objective.

Incident and runbook toil

6. Rundeck and PagerDuty Runbook Automation

Rundeck turns runbook steps into jobs with options, access control and an audit log. PagerDuty acquired Rundeck in 2020 and sells the commercial product as PagerDuty Runbook Automation; Rundeck's open-source edition remains available. Typical jobs collect diagnostics, restart a stuck consumer, clear a cache or fail over a database. Support engineers can run approved jobs themselves instead of paging an SRE, and jobs can be triggered from incidents once they are trusted.

Move a runbook through stages: a written document, then a script, then a Rundeck job that people click, and only then an automatic trigger. Each stage proves the steps are safe before less human judgement is involved. Our article on automating incident response with runbooks covers the process side.

Capacity and certificate toil

7. KEDA

KEDA (Kubernetes Event-driven Autoscaling), a CNCF graduated project, scales workloads on external signals such as queue length, Kafka consumer lag or a Prometheus query, and can scale to zero. It removes the manual job of adding consumers when a backlog builds.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    name: order-worker
  minReplicaCount: 1
  maxReplicaCount: 30
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(orders_queue_depth)
        threshold: "100"

8. cert-manager

Expired certificates still cause outages. cert-manager, a CNCF graduated project, issues certificates from ACME providers such as Let's Encrypt, from HashiCorp Vault or from private CAs, stores them as Kubernetes Secrets, and renews them before expiry.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shop-tls
  namespace: shop
spec:
  secretName: shop-tls
  dnsNames:
    - shop.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Alert on the certmanager_certificate_expiration_timestamp_seconds metric as a safety net in case renewal fails.

Resilience testing toil

9. LitmusChaos

LitmusChaos, a CNCF incubating project, runs chaos experiments on Kubernetes: pod deletion, network latency, CPU stress, node drain and more. Experiments can include probes that check your SLIs during the fault, and they can run on a schedule. That replaces hand-run failover drills with repeatable tests whose results are recorded. Start in staging, keep the blast radius small, and stop experiments automatically if probes fail. See how chaos engineering improves resilience.

10. Grafana k6

k6 is an open-source load testing tool from Grafana Labs. Tests are JavaScript files, and thresholds turn results into a pass or fail that CI can act on, so capacity checks run on every release instead of once before a launch.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 50,
  duration: '5m',
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<400'],
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/health');
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Run it with k6 run load.js; a failed threshold returns a non-zero exit code and fails the pipeline.

Measuring whether automation reduced toil

The goal is less manual work and fewer incidents, so track a few numbers before and after each change. They show the effect and help you decide what to automate next.

MeasureHow to collect itWhat improvement looks like
Hours of toil per engineer per weekToil log or ticket labelsFalls after each automation is adopted
Pages per on-call shiftPaging tool reportsFewer, more actionable pages after Alertmanager and SLO alert changes
Time to restore serviceIncident recordsShorter when runbook jobs replace manual steps
Change failure rateDeploy and incident recordsLower with GitOps, dry runs and load test gates
Manual production changesCloud audit logs, Argo CD self-heal eventsClose to zero outside break-glass use
Certificate or capacity incidentsIncident tagsNone after cert-manager and KEDA are in place

Time to restore and change failure rate are two of the DORA delivery metrics, so the same data supports wider DevOps reporting. Incident data usually lives in your paging or incident management tool.

How these tools connect

Terraform creates the cluster and cloud resources. Argo CD deploys cert-manager, KEDA, Prometheus and the applications from Git. Sloth turns SLO specs, also in Git, into rules that Prometheus evaluates. Alertmanager groups and routes the resulting alerts, and the paging tool attaches links to Rundeck jobs for the known fixes. k6 and LitmusChaos run from the same CI system that deploys, so a release that breaks a threshold or fails an experiment does not reach production. Ansible covers the virtual machines and appliances outside Kubernetes. Because every piece is defined in a repository, the whole reliability setup can be reviewed, versioned and rebuilt.

Guardrails for SRE automation

GuardrailWhyExample
Rate limits and batch sizesStops automation from changing everything at onceAnsible serial, Argo CD sync waves
Audit logsYou must know what automation did during an incidentRundeck execution history, Git history for GitOps
Kill switchHumans need to pause automation fastDisable Argo CD auto-sync, pause a Rundeck schedule
Dry runsSee the change before it happensterraform plan, ansible-playbook --check --diff
BoundsPrevent runaway scaling or costKEDA maxReplicaCount
OwnershipAutomation without an owner rotsCode owners on automation repositories

A 90-day adoption plan

  1. Weeks 1 to 2: keep a toil log across the team and score tasks.
  2. Weeks 3 to 6: fix the noisiest alerting with Alertmanager grouping and inhibition, and generate SLO alerts with Sloth for your top two services.
  3. Weeks 5 to 8: put cert-manager and Argo CD in place for Kubernetes services, and Terraform for new infrastructure.
  4. Weeks 7 to 10: convert the five most-used runbooks into Rundeck jobs.
  5. Weeks 9 to 12: add k6 thresholds to release pipelines and one scheduled LitmusChaos experiment in staging. Re-measure toil and compare with week 2.

Learn SRE automation hands-on

Prepare for interviews with our 100+ SRE interview questions. The Prometheus and Grafana training covers alerting and SLO dashboards, and the DevOps training covers Terraform, Ansible, Kubernetes and GitOps. Both run in Kharadi, Pune and live online.

FAQ

What is toil in SRE?

Google's SRE book defines toil as operational work that is manual, repetitive, automatable, tactical, without lasting value, and that grows linearly with the service. Examples include manual restarts, certificate renewals and hand-run deployments.

Which SRE automation tool should I learn first?

Learn Terraform and Ansible first, because they appear in most SRE and DevOps roles. Then add Prometheus with Alertmanager for alerting and Argo CD if you work with Kubernetes.

How is this list different from a list of SRE tools?

This list focuses on automation that removes specific manual tasks. Broader SRE tool lists include monitoring, tracing and incident management platforms; see our top 10 SRE tools post for that view.

What does Sloth do?

Sloth reads a short SLO specification and generates Prometheus recording rules and multi-window burn-rate alerts. It saves writing and maintaining those rules by hand for every service.

Is Rundeck still open source?

Yes. Rundeck has an open-source edition. PagerDuty, which acquired Rundeck in 2020, sells the commercial version as PagerDuty Runbook Automation with extra enterprise features.

Should runbooks be fully automated?

Only after the steps are proven safe. Move from a written runbook to a script, then a self-service job, and finally an alert-triggered job, adding logs, limits and a way to stop it at each stage.

How does chaos engineering reduce toil?

Scheduled experiments with tools such as LitmusChaos replace manual failover drills and catch weaknesses before they cause incidents, which reduces the unplanned work that follows outages.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.