15 DevOps Metrics That Predict System Failure
15 leading-indicator metrics that warn of system failure: burn rate, p99 latency, saturation, queue lag, cert expiry and DORA instability metrics.
The metrics that best predict system failure are leading indicators: signals that change before users feel an outage. Watch error budget burn rate, tail latency, saturation of CPU, memory, disks and connection pools, queue backlogs, retries, replication lag and certificate expiry, plus delivery metrics such as change fail rate that show when releases are making the system fragile.
Most dashboards are full of lagging indicators, which tell you an outage happened. This guide picks 15 metrics that tend to move first, explains what each one warns about, and gives example PromQL where it helps. The Prometheus functions used here were checked against the current Prometheus documentation, and delivery metric names follow DORA's current five-metric model. It sits in our monitoring and observability cluster; for performance metrics aimed at team improvement rather than failure prediction, see key DevOps metrics used by high-performing teams.
Key takeaways
- Leading indicators trend before an outage; lagging ones confirm it. Alert on trends and rates of change, not only on fixed thresholds.
- Saturation is the golden signal most linked to future failure. Google's SRE book defines it as how "full" a service is.
- PromQL's
predict_linear()turns a growth trend into a time-to-exhaustion alert. - Error budget burn rate warns early: at a burn rate of 14.4, a 99.9% SLO loses 2% of its monthly budget in one hour.
- DORA's instability metrics, change fail rate and deployment rework rate, show when your release process is creating risk.
- Pick a few metrics per service and act on them. Fifteen alerts nobody reads predict nothing.
The 15 metrics at a glance
| # | Metric | Category | Warns about |
|---|---|---|---|
| 1 | Error budget burn rate | Reliability | SLO breach in hours or days |
| 2 | Tail latency (p99) | User experience | Contention, slow dependencies |
| 3 | Error rate and error latency | User experience | Partial failure spreading |
| 4 | CPU saturation | Saturation | Throughput collapse under load |
| 5 | Memory headroom | Saturation | Out-of-memory kills |
| 6 | Container restart rate | Stability | Crash loops, leaks |
| 7 | Disk time to full | Saturation | Write failures |
| 8 | Connection pool usage | Saturation | Requests queuing for connections |
| 9 | Queue depth and consumer lag | Saturation | Backlog that never drains |
| 10 | Retry and timeout rate | Dependencies | Retry storms, cascading failure |
| 11 | Replication lag | Data | Stale reads, data loss on failover |
| 12 | Certificate days to expiry | Configuration | Sudden TLS outage |
| 13 | Change fail rate | Delivery | Fragile release process |
| 14 | Deployment rework rate | Delivery | Unplanned fixes after incidents |
| 15 | Failed deployment recovery time | Delivery | Long outages when releases go wrong |
Leading and lagging indicators
A lagging indicator, such as total downtime last month, tells you how bad things were. A leading indicator, such as a disk that is 2% fuller every hour, tells you how bad things are about to be. Prediction depends on leading indicators and on looking at their direction and speed, not just their current value. A disk at 80% that has been flat for a year is less worrying than one at 60% that grew 10 points since yesterday.
Google's SRE book recommends spending more effort on symptoms than on causes when paging people. Leading indicators are mostly causes, so use them carefully: page only when the outcome is near and certain (a disk that will fill in two hours), and send the rest to a ticket or dashboard.
Reliability and user experience signals
1. Error budget burn rate
Burn rate measures how fast a service is consuming its error budget relative to its SLO. A rate of 1 spends exactly the full budget over the SLO window. Google's SRE workbook suggests paging for a 99.9% SLO when the burn rate is 14.4 over one hour (2% of a 30-day budget gone) and opening a ticket when it is 1 over three days. A slow burn is one of the clearest early warnings you can have: users are only slightly affected today, but the trend will breach the objective if nothing changes. Our guide to monitoring strategies for zero downtime shows the alert rules.
2. Tail latency (p99)
Average latency hides trouble. When a dependency slows down or a lock becomes contended, the slowest 1% of requests degrade first while the average barely moves. Track p99 from histograms and watch its trend across days and releases.
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))
3. Error rate, with error latency tracked separately
A small, steady rise in 5xx responses often comes before a larger failure, for example as one replica or one dependency starts to fail. The SRE book also advises tracking the latency of failed requests separately, since fast errors can make overall latency look better during an incident. A falling average latency combined with a rising error rate is a warning, not good news.
Saturation signals
Saturation is the golden signal most closely linked to future failure, because systems often degrade sharply as they approach full capacity. Brendan Gregg's USE method gives a checklist: for every resource, check utilization, saturation (queued work) and errors.
4. CPU saturation
High CPU utilization alone is not a problem; queued work waiting for CPU is. In containers, also watch CPU throttling against limits, which adds latency even when the node has spare capacity. A sustained rise in throttling or run queue length predicts latency problems at the next traffic peak.
5. Memory headroom
Compare a container's working memory with its limit. A slow, steady climb that never drops after garbage collection suggests a leak, and it predicts an out-of-memory kill at a time you can estimate. Alert on the trend, and on OOM-killed container terminations, which are the confirmation.
6. Container restart rate
Restarts that rise slowly over days often point to leaks, failing health checks or intermittent dependency problems. In Kubernetes with kube-state-metrics, the restart counter per container makes this easy to track:
increase(kube_pod_container_status_restarts_total[1h]) > 3
The increase() function computes how much a counter grew over the range, which suits this kind of counter.
7. Disk time to full
Disks rarely fill without warning; they grow at a measurable rate. PromQL's predict_linear() uses linear regression over a range to predict a value some seconds ahead, which turns that growth into a time-to-full alert.
predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 24 * 3600) < 0
This expression is true for filesystems predicted to run out of space within 24 hours. Use a shorter horizon with a page, and a longer one with a ticket.
8. Connection pool usage
Database and HTTP client connection pools have fixed sizes. As usage approaches the maximum, requests start waiting for a free connection, latency rises and timeouts follow. Track active connections against the pool size and the time requests wait to get one. Pool exhaustion often appears at the database too, as connections near the server's configured limit.
9. Queue depth and consumer lag
For asynchronous systems, the backlog is the key signal. A queue that grows during a peak and drains afterwards is fine. A queue whose minimum level rises day after day means consumers cannot keep up, and eventually messages will expire, or the queue will hit its limits. Track depth, the age of the oldest message and consumer lag, and alert on sustained growth.
Dependency, data and configuration signals
10. Retry and timeout rate
Retries hide small failures, which is useful until they multiply load on a struggling dependency and cause a cascading failure. A rising retry rate or timeout rate on calls to a dependency is an early sign that the dependency is degrading, even while the success rate seen by users still looks fine. Track retries separately from first attempts.
11. Replication lag
Database and cache replicas that fall behind serve stale data, and if the primary fails while lag is high, failing over can lose recent writes. Lag that keeps growing also signals that a replica cannot keep up with write volume. Alert when lag exceeds what your recovery point objective allows.
12. Certificate days to expiry
An expired TLS certificate causes an instant, total outage for the affected endpoint, and the date is known in advance. Monitor the days remaining on every public and internal certificate, including those used between services, and alert with enough lead time to fix automated renewal if it has failed. External probes, such as the Prometheus Blackbox Exporter, can report certificate expiry for the endpoints they check.
Delivery signals
Changes are a frequent trigger for incidents, so the health of the release process predicts failure too. DORA's current model has five software delivery metrics: change lead time, deployment frequency and failed deployment recovery time for throughput, and change fail rate and deployment rework rate for instability. The three below are the most useful for failure prediction. Our post on the value of DORA metrics covers the full model.
13. Change fail rate
DORA defines change fail rate as the share of deployments that need immediate intervention, such as a rollback or hotfix. A rising change fail rate means more of your releases are causing problems, and more future incidents are likely. Look for causes in test coverage, review practices and batch size.
14. Deployment rework rate
DORA's newer instability metric measures the share of deployments that are unplanned and triggered by a production incident. When rework rises, the team is spending more of its release capacity fixing production rather than delivering planned work, which usually means the system has underlying weaknesses.
15. Failed deployment recovery time
This DORA metric, which replaced the older MTTR measure in DORA's model, is how long it takes to recover from a deployment that needs immediate intervention. It does not predict whether a failure will happen, but it predicts how much a failure will hurt. If recovery time is creeping up, rollback paths, alerting or runbooks need attention before the next bad release.
Turning metrics into predictions
- Choose per service. Pick three to six of these metrics that match how each service fails. A queue worker needs metric 9; a stateless API may not.
- Baseline first. Collect at least a few weeks of data so you know normal daily and weekly patterns.
- Alert on trends. Use rates of change,
predict_linear()and burn rates rather than fixed thresholds alone. - Route by urgency. Page for imminent, certain outcomes; open tickets for slow trends.
- Review after incidents. In each postmortem, ask which metric moved first and whether an alert would have caught it.
For more signals to collect, read our observability metrics for cloud-native systems, and for alert setup, monitoring alerts to configure in cloud. Our Prometheus and Grafana training, in Kharadi, Pune and live online across India, covers PromQL and alerting hands-on.
Official references
- Google SRE book: Monitoring distributed systems
- Google SRE workbook: Alerting on SLOs
- Prometheus: Query functions
- DORA: Software delivery performance metrics
- Brendan Gregg: The USE Method
FAQs
What is a leading indicator of system failure?
A metric that changes before users feel an outage, such as a disk filling at a steady rate, rising tail latency, growing queue backlog or a slow error budget burn. It gives you time to act.
Which golden signal best predicts failure?
Saturation, because systems often degrade sharply as they approach full capacity. Track how full CPU, memory, disks, connection pools and queues are, and how fast they are filling.
How does predict_linear work in Prometheus?
It applies simple linear regression to a range of samples and predicts the value a given number of seconds ahead. It is commonly used to alert before a disk or quota runs out.
What are DORA's five software delivery metrics?
Change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. The last two measure instability.
What is deployment rework rate?
The share of deployments that are unplanned and triggered by a production incident. A rising value means more release effort goes into fixing production problems.
How many failure-prediction alerts should a service have?
Only as many as people will act on. Start with three to six leading indicators matched to how the service fails, page only for imminent problems and send slow trends to tickets.
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0