10 SRE Automation Tools That Cut Toil
10 SRE automation tools mapped to the toil they remove: Terraform, Ansible, Argo CD, Alertmanager, Sloth, Rundeck, KEDA, cert-manager, Litmus, k6.
Ten automation tools that remove the most toil for SRE teams are Terraform, Ansible, Argo CD, Prometheus Alertmanager, Sloth, Rundeck (PagerDuty Runbook Automation), KEDA, cert-manager, LitmusChaos and Grafana k6. Each one replaces a repetitive manual job: provisioning, fixing drift, deploying, routing alerts, writing SLO alerts, running runbooks, scaling, renewing certificates, testing failure and testing load.
This post is organised by the toil each tool removes, with a small working example for each. It is not a general catalogue of SRE software. For the broader list of monitoring, incident and observability platforms, see our companion post on the top 10 SRE tools. For what the SRE role covers, start with the role of SREs in DevOps teams. Tool features were checked against official project documentation in November 2026.
Key takeaways
- Google's SRE book defines toil as manual, repetitive, automatable operational work with no lasting value that grows with the service. Automate the toil you can measure first.
- Keep a toil log for two weeks: what you did by hand, how often, and how long it took. It tells you which tool to adopt.
- Declarative tools (Terraform, Argo CD, cert-manager, KEDA) remove whole classes of toil because a controller keeps reconciling to the desired state.
- Generate SLO alerts from a spec with Sloth instead of hand-writing burn-rate rules for each service.
- Automate runbooks step by step: document, script, add a button, then trigger from alerts once the steps are proven safe.
- Every automation needs logs, limits and an off switch. Unbounded automation can turn a small incident into a large one.
Toil, and how to choose what to automate
The Google SRE book describes toil as work tied to running a production service that is manual, repetitive, automatable, tactical, without enduring value, and that scales linearly as the service grows. Its guidance is to keep an SRE's operational work, toil included, under half of their time so the rest goes to engineering that reduces future toil.
A practical way to choose: score each manual task by frequency multiplied by time spent, then weight up anything that is risky when done by hand (production changes at 3 a.m.) or that delays other teams (access requests, environment creation). The highest scores are your first automation targets. The table below maps common toil to the tool in this list that addresses it.
| Recurring manual task | Tool | Automation style |
|---|---|---|
| Creating and changing cloud infrastructure by console | Terraform / OpenTofu | Declarative plan and apply from Git |
| Fixing configuration drift on servers | Ansible | Idempotent playbooks, scheduled or triggered |
| Running deploy scripts and fixing cluster drift | Argo CD | GitOps reconciliation |
| Sorting duplicate alerts during an incident | Prometheus Alertmanager | Grouping, inhibition, routing |
| Writing burn-rate alert rules per service | Sloth | Generate rules from an SLO spec |
| Running the same diagnostic and restart steps | Rundeck / PagerDuty Runbook Automation | Self-service and alert-triggered jobs |
| Adjusting replicas for queue backlogs | KEDA | Event-driven autoscaling |
| Renewing TLS certificates | cert-manager | Automatic issuance and renewal |
| Manual failover drills | LitmusChaos | Scheduled chaos experiments |
| Ad hoc load tests before launches | Grafana k6 | Load tests with pass/fail thresholds in CI |
Infrastructure and configuration toil
1. Terraform (and OpenTofu)
Terraform describes infrastructure in HCL, shows a plan of changes, and applies them through provider APIs. For SREs the toil reduction is consistency: environments are created the same way every time, changes are reviewed in pull requests, and terraform plan run on a schedule reveals drift made by hand in a console. HashiCorp licenses Terraform under the Business Source License; OpenTofu is the open-source fork with the same workflow.
# nightly drift check in CI: exit code 2 means changes are pending
terraform init -input=false
terraform plan -detailed-exitcode -input=false -lock=false
# 0 = no changes, 1 = error, 2 = drift or pending changes
2. Ansible
Ansible applies idempotent playbooks over SSH with no agent. SRE teams use it for OS patching in batches, rotating configuration, restarting services in a controlled order, and repairing drift found by monitoring. The serial keyword limits how many hosts change at once, and max_fail_percentage stops a rollout that is going wrong.
- name: Rolling patch of web tier
hosts: web
become: true
serial: "20%"
max_fail_percentage: 10
tasks:
- name: Apply security updates
ansible.builtin.dnf:
name: "*"
security: true
state: latest
- name: Reboot if required
ansible.builtin.reboot:
reboot_timeout: 600
3. Argo CD
Argo CD watches Git and keeps Kubernetes clusters in the declared state. Deploys stop being scripts someone runs; they are merges. With automated sync and self-heal enabled, a manual change in the cluster is reverted to what Git says, which removes a whole category of "who changed this?" investigations. Rollback is a Git revert.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: checkout
namespace: argocd
spec:
project: default
source:
repoURL: https://git.example.com/platform/apps
path: checkout/overlays/prod
targetRevision: main
destination:
server: https://kubernetes.default.svc
namespace: checkout
syncPolicy:
automated:
prune: true
selfHeal: true
Read more on how GitOps works.
Alerting and SLO toil
4. Prometheus Alertmanager
Alertmanager receives alerts from Prometheus and decides what reaches a human. Grouping turns fifty pod alerts into one notification. Inhibition suppresses symptom alerts while a cause alert is firing, for example service alerts while the whole cluster is down. Silences mute known work during maintenance. Routing sends each alert to the right team and channel. All of this cuts the manual triage that wears out on-call engineers.
route:
receiver: default
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: ['severity="page"']
receiver: oncall-pager
inhibit_rules:
- source_matchers: ['alertname="ClusterDown"']
target_matchers: ['severity="warning"']
equal: ['cluster']
receivers:
- name: default
- name: oncall-pager
Pair it with Grafana dashboards as described in using Prometheus and Grafana together.
5. Sloth
Writing multi-window, multi-burn-rate alerts by hand for each service is slow and error prone. Sloth takes a short SLO specification and generates the Prometheus recording rules and alerting rules for you, following the burn-rate approach from the Google SRE Workbook. It can run as a CLI in CI or as a Kubernetes controller.
version: "prometheus/v1"
service: "checkout"
slos:
- name: "requests-availability"
objective: 99.9
description: "Checkout API non-5xx responses"
sli:
events:
error_query: sum(rate(http_requests_total{job="checkout",code=~"5.."}[{{.window}}]))
total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
alerting:
name: CheckoutHighErrorBudgetBurn
page_alert:
labels:
severity: page
ticket_alert:
labels:
severity: ticket
Pyrra is a similar open-source project if you prefer a UI. Our post on SLOs and business goals explains how to choose the objective.
Incident and runbook toil
6. Rundeck and PagerDuty Runbook Automation
Rundeck turns runbook steps into jobs with options, access control and an audit log. PagerDuty acquired Rundeck in 2020 and sells the commercial product as PagerDuty Runbook Automation; Rundeck's open-source edition remains available. Typical jobs collect diagnostics, restart a stuck consumer, clear a cache or fail over a database. Support engineers can run approved jobs themselves instead of paging an SRE, and jobs can be triggered from incidents once they are trusted.
Move a runbook through stages: a written document, then a script, then a Rundeck job that people click, and only then an automatic trigger. Each stage proves the steps are safe before less human judgement is involved. Our article on automating incident response with runbooks covers the process side.
Capacity and certificate toil
7. KEDA
KEDA (Kubernetes Event-driven Autoscaling), a CNCF graduated project, scales workloads on external signals such as queue length, Kafka consumer lag or a Prometheus query, and can scale to zero. It removes the manual job of adding consumers when a backlog builds.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-worker
spec:
scaleTargetRef:
name: order-worker
minReplicaCount: 1
maxReplicaCount: 30
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
query: sum(orders_queue_depth)
threshold: "100"
8. cert-manager
Expired certificates still cause outages. cert-manager, a CNCF graduated project, issues certificates from ACME providers such as Let's Encrypt, from HashiCorp Vault or from private CAs, stores them as Kubernetes Secrets, and renews them before expiry.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: shop-tls
namespace: shop
spec:
secretName: shop-tls
dnsNames:
- shop.example.com
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
Alert on the certmanager_certificate_expiration_timestamp_seconds metric as a safety net in case renewal fails.
Resilience testing toil
9. LitmusChaos
LitmusChaos, a CNCF incubating project, runs chaos experiments on Kubernetes: pod deletion, network latency, CPU stress, node drain and more. Experiments can include probes that check your SLIs during the fault, and they can run on a schedule. That replaces hand-run failover drills with repeatable tests whose results are recorded. Start in staging, keep the blast radius small, and stop experiments automatically if probes fail. See how chaos engineering improves resilience.
10. Grafana k6
k6 is an open-source load testing tool from Grafana Labs. Tests are JavaScript files, and thresholds turn results into a pass or fail that CI can act on, so capacity checks run on every release instead of once before a launch.
import http from 'k6/http';
import { check } from 'k6';
export const options = {
vus: 50,
duration: '5m',
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<400'],
},
};
export default function () {
const res = http.get('https://staging.example.com/api/health');
check(res, { 'status is 200': (r) => r.status === 200 });
}
Run it with k6 run load.js; a failed threshold returns a non-zero exit code and fails the pipeline.
Measuring whether automation reduced toil
The goal is less manual work and fewer incidents, so track a few numbers before and after each change. They show the effect and help you decide what to automate next.
| Measure | How to collect it | What improvement looks like |
|---|---|---|
| Hours of toil per engineer per week | Toil log or ticket labels | Falls after each automation is adopted |
| Pages per on-call shift | Paging tool reports | Fewer, more actionable pages after Alertmanager and SLO alert changes |
| Time to restore service | Incident records | Shorter when runbook jobs replace manual steps |
| Change failure rate | Deploy and incident records | Lower with GitOps, dry runs and load test gates |
| Manual production changes | Cloud audit logs, Argo CD self-heal events | Close to zero outside break-glass use |
| Certificate or capacity incidents | Incident tags | None after cert-manager and KEDA are in place |
Time to restore and change failure rate are two of the DORA delivery metrics, so the same data supports wider DevOps reporting. Incident data usually lives in your paging or incident management tool.
How these tools connect
Terraform creates the cluster and cloud resources. Argo CD deploys cert-manager, KEDA, Prometheus and the applications from Git. Sloth turns SLO specs, also in Git, into rules that Prometheus evaluates. Alertmanager groups and routes the resulting alerts, and the paging tool attaches links to Rundeck jobs for the known fixes. k6 and LitmusChaos run from the same CI system that deploys, so a release that breaks a threshold or fails an experiment does not reach production. Ansible covers the virtual machines and appliances outside Kubernetes. Because every piece is defined in a repository, the whole reliability setup can be reviewed, versioned and rebuilt.
Guardrails for SRE automation
| Guardrail | Why | Example |
|---|---|---|
| Rate limits and batch sizes | Stops automation from changing everything at once | Ansible serial, Argo CD sync waves |
| Audit logs | You must know what automation did during an incident | Rundeck execution history, Git history for GitOps |
| Kill switch | Humans need to pause automation fast | Disable Argo CD auto-sync, pause a Rundeck schedule |
| Dry runs | See the change before it happens | terraform plan, ansible-playbook --check --diff |
| Bounds | Prevent runaway scaling or cost | KEDA maxReplicaCount |
| Ownership | Automation without an owner rots | Code owners on automation repositories |
A 90-day adoption plan
- Weeks 1 to 2: keep a toil log across the team and score tasks.
- Weeks 3 to 6: fix the noisiest alerting with Alertmanager grouping and inhibition, and generate SLO alerts with Sloth for your top two services.
- Weeks 5 to 8: put cert-manager and Argo CD in place for Kubernetes services, and Terraform for new infrastructure.
- Weeks 7 to 10: convert the five most-used runbooks into Rundeck jobs.
- Weeks 9 to 12: add k6 thresholds to release pipelines and one scheduled LitmusChaos experiment in staging. Re-measure toil and compare with week 2.
Learn SRE automation hands-on
Prepare for interviews with our 100+ SRE interview questions. The Prometheus and Grafana training covers alerting and SLO dashboards, and the DevOps training covers Terraform, Ansible, Kubernetes and GitOps. Both run in Kharadi, Pune and live online.
FAQ
What is toil in SRE?
Google's SRE book defines toil as operational work that is manual, repetitive, automatable, tactical, without lasting value, and that grows linearly with the service. Examples include manual restarts, certificate renewals and hand-run deployments.
Which SRE automation tool should I learn first?
Learn Terraform and Ansible first, because they appear in most SRE and DevOps roles. Then add Prometheus with Alertmanager for alerting and Argo CD if you work with Kubernetes.
How is this list different from a list of SRE tools?
This list focuses on automation that removes specific manual tasks. Broader SRE tool lists include monitoring, tracing and incident management platforms; see our top 10 SRE tools post for that view.
What does Sloth do?
Sloth reads a short SLO specification and generates Prometheus recording rules and multi-window burn-rate alerts. It saves writing and maintaining those rules by hand for every service.
Is Rundeck still open source?
Yes. Rundeck has an open-source edition. PagerDuty, which acquired Rundeck in 2020, sells the commercial version as PagerDuty Runbook Automation with extra enterprise features.
Should runbooks be fully automated?
Only after the steps are proven safe. Move from a written runbook to a script, then a self-service job, and finally an alert-triggered job, adding logs, limits and a way to stop it at each stage.
How does chaos engineering reduce toil?
Scheduled experiments with tools such as LitmusChaos replace manual failover drills and catch weaknesses before they cause incidents, which reduces the unplanned work that follows outages.
References
- Google SRE book: Eliminating Toil
- Google SRE Workbook: Alerting on SLOs
- Terraform: plan command
- Ansible: Controlling playbook execution (serial, max_fail_percentage)
- Argo CD: Automated sync policy
- Prometheus: Alertmanager configuration
- Sloth documentation
- Rundeck documentation
- KEDA: Prometheus scaler
- cert-manager: Certificate resource
- LitmusChaos documentation
- Grafana k6: Thresholds
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0