15 DevOps Metrics That Predict System Failure

15 leading-indicator metrics that warn of system failure: burn rate, p99 latency, saturation, queue lag, cert expiry and DORA instability metrics.

Dec 29, 2025 - 14:53
Nov 13, 2026 - 23:42
 0  54
15 DevOps Metrics That Predict System Failure

The metrics that best predict system failure are leading indicators: signals that change before users feel an outage. Watch error budget burn rate, tail latency, saturation of CPU, memory, disks and connection pools, queue backlogs, retries, replication lag and certificate expiry, plus delivery metrics such as change fail rate that show when releases are making the system fragile.

Most dashboards are full of lagging indicators, which tell you an outage happened. This guide picks 15 metrics that tend to move first, explains what each one warns about, and gives example PromQL where it helps. The Prometheus functions used here were checked against the current Prometheus documentation, and delivery metric names follow DORA's current five-metric model. It sits in our monitoring and observability cluster; for performance metrics aimed at team improvement rather than failure prediction, see key DevOps metrics used by high-performing teams.

Key takeaways

  • Leading indicators trend before an outage; lagging ones confirm it. Alert on trends and rates of change, not only on fixed thresholds.
  • Saturation is the golden signal most linked to future failure. Google's SRE book defines it as how "full" a service is.
  • PromQL's predict_linear() turns a growth trend into a time-to-exhaustion alert.
  • Error budget burn rate warns early: at a burn rate of 14.4, a 99.9% SLO loses 2% of its monthly budget in one hour.
  • DORA's instability metrics, change fail rate and deployment rework rate, show when your release process is creating risk.
  • Pick a few metrics per service and act on them. Fifteen alerts nobody reads predict nothing.

The 15 metrics at a glance

#MetricCategoryWarns about
1Error budget burn rateReliabilitySLO breach in hours or days
2Tail latency (p99)User experienceContention, slow dependencies
3Error rate and error latencyUser experiencePartial failure spreading
4CPU saturationSaturationThroughput collapse under load
5Memory headroomSaturationOut-of-memory kills
6Container restart rateStabilityCrash loops, leaks
7Disk time to fullSaturationWrite failures
8Connection pool usageSaturationRequests queuing for connections
9Queue depth and consumer lagSaturationBacklog that never drains
10Retry and timeout rateDependenciesRetry storms, cascading failure
11Replication lagDataStale reads, data loss on failover
12Certificate days to expiryConfigurationSudden TLS outage
13Change fail rateDeliveryFragile release process
14Deployment rework rateDeliveryUnplanned fixes after incidents
15Failed deployment recovery timeDeliveryLong outages when releases go wrong

Leading and lagging indicators

A lagging indicator, such as total downtime last month, tells you how bad things were. A leading indicator, such as a disk that is 2% fuller every hour, tells you how bad things are about to be. Prediction depends on leading indicators and on looking at their direction and speed, not just their current value. A disk at 80% that has been flat for a year is less worrying than one at 60% that grew 10 points since yesterday.

Google's SRE book recommends spending more effort on symptoms than on causes when paging people. Leading indicators are mostly causes, so use them carefully: page only when the outcome is near and certain (a disk that will fill in two hours), and send the rest to a ticket or dashboard.

Reliability and user experience signals

1. Error budget burn rate

Burn rate measures how fast a service is consuming its error budget relative to its SLO. A rate of 1 spends exactly the full budget over the SLO window. Google's SRE workbook suggests paging for a 99.9% SLO when the burn rate is 14.4 over one hour (2% of a 30-day budget gone) and opening a ticket when it is 1 over three days. A slow burn is one of the clearest early warnings you can have: users are only slightly affected today, but the trend will breach the objective if nothing changes. Our guide to monitoring strategies for zero downtime shows the alert rules.

2. Tail latency (p99)

Average latency hides trouble. When a dependency slows down or a lock becomes contended, the slowest 1% of requests degrade first while the average barely moves. Track p99 from histograms and watch its trend across days and releases.

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))

3. Error rate, with error latency tracked separately

A small, steady rise in 5xx responses often comes before a larger failure, for example as one replica or one dependency starts to fail. The SRE book also advises tracking the latency of failed requests separately, since fast errors can make overall latency look better during an incident. A falling average latency combined with a rising error rate is a warning, not good news.

Saturation signals

Saturation is the golden signal most closely linked to future failure, because systems often degrade sharply as they approach full capacity. Brendan Gregg's USE method gives a checklist: for every resource, check utilization, saturation (queued work) and errors.

4. CPU saturation

High CPU utilization alone is not a problem; queued work waiting for CPU is. In containers, also watch CPU throttling against limits, which adds latency even when the node has spare capacity. A sustained rise in throttling or run queue length predicts latency problems at the next traffic peak.

5. Memory headroom

Compare a container's working memory with its limit. A slow, steady climb that never drops after garbage collection suggests a leak, and it predicts an out-of-memory kill at a time you can estimate. Alert on the trend, and on OOM-killed container terminations, which are the confirmation.

6. Container restart rate

Restarts that rise slowly over days often point to leaks, failing health checks or intermittent dependency problems. In Kubernetes with kube-state-metrics, the restart counter per container makes this easy to track:

increase(kube_pod_container_status_restarts_total[1h]) > 3

The increase() function computes how much a counter grew over the range, which suits this kind of counter.

7. Disk time to full

Disks rarely fill without warning; they grow at a measurable rate. PromQL's predict_linear() uses linear regression over a range to predict a value some seconds ahead, which turns that growth into a time-to-full alert.

predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 24 * 3600) < 0

This expression is true for filesystems predicted to run out of space within 24 hours. Use a shorter horizon with a page, and a longer one with a ticket.

8. Connection pool usage

Database and HTTP client connection pools have fixed sizes. As usage approaches the maximum, requests start waiting for a free connection, latency rises and timeouts follow. Track active connections against the pool size and the time requests wait to get one. Pool exhaustion often appears at the database too, as connections near the server's configured limit.

9. Queue depth and consumer lag

For asynchronous systems, the backlog is the key signal. A queue that grows during a peak and drains afterwards is fine. A queue whose minimum level rises day after day means consumers cannot keep up, and eventually messages will expire, or the queue will hit its limits. Track depth, the age of the oldest message and consumer lag, and alert on sustained growth.

Dependency, data and configuration signals

10. Retry and timeout rate

Retries hide small failures, which is useful until they multiply load on a struggling dependency and cause a cascading failure. A rising retry rate or timeout rate on calls to a dependency is an early sign that the dependency is degrading, even while the success rate seen by users still looks fine. Track retries separately from first attempts.

11. Replication lag

Database and cache replicas that fall behind serve stale data, and if the primary fails while lag is high, failing over can lose recent writes. Lag that keeps growing also signals that a replica cannot keep up with write volume. Alert when lag exceeds what your recovery point objective allows.

12. Certificate days to expiry

An expired TLS certificate causes an instant, total outage for the affected endpoint, and the date is known in advance. Monitor the days remaining on every public and internal certificate, including those used between services, and alert with enough lead time to fix automated renewal if it has failed. External probes, such as the Prometheus Blackbox Exporter, can report certificate expiry for the endpoints they check.

Delivery signals

Changes are a frequent trigger for incidents, so the health of the release process predicts failure too. DORA's current model has five software delivery metrics: change lead time, deployment frequency and failed deployment recovery time for throughput, and change fail rate and deployment rework rate for instability. The three below are the most useful for failure prediction. Our post on the value of DORA metrics covers the full model.

13. Change fail rate

DORA defines change fail rate as the share of deployments that need immediate intervention, such as a rollback or hotfix. A rising change fail rate means more of your releases are causing problems, and more future incidents are likely. Look for causes in test coverage, review practices and batch size.

14. Deployment rework rate

DORA's newer instability metric measures the share of deployments that are unplanned and triggered by a production incident. When rework rises, the team is spending more of its release capacity fixing production rather than delivering planned work, which usually means the system has underlying weaknesses.

15. Failed deployment recovery time

This DORA metric, which replaced the older MTTR measure in DORA's model, is how long it takes to recover from a deployment that needs immediate intervention. It does not predict whether a failure will happen, but it predicts how much a failure will hurt. If recovery time is creeping up, rollback paths, alerting or runbooks need attention before the next bad release.

Turning metrics into predictions

  1. Choose per service. Pick three to six of these metrics that match how each service fails. A queue worker needs metric 9; a stateless API may not.
  2. Baseline first. Collect at least a few weeks of data so you know normal daily and weekly patterns.
  3. Alert on trends. Use rates of change, predict_linear() and burn rates rather than fixed thresholds alone.
  4. Route by urgency. Page for imminent, certain outcomes; open tickets for slow trends.
  5. Review after incidents. In each postmortem, ask which metric moved first and whether an alert would have caught it.

For more signals to collect, read our observability metrics for cloud-native systems, and for alert setup, monitoring alerts to configure in cloud. Our Prometheus and Grafana training, in Kharadi, Pune and live online across India, covers PromQL and alerting hands-on.

Official references

FAQs

What is a leading indicator of system failure?

A metric that changes before users feel an outage, such as a disk filling at a steady rate, rising tail latency, growing queue backlog or a slow error budget burn. It gives you time to act.

Which golden signal best predicts failure?

Saturation, because systems often degrade sharply as they approach full capacity. Track how full CPU, memory, disks, connection pools and queues are, and how fast they are filling.

How does predict_linear work in Prometheus?

It applies simple linear regression to a range of samples and predicts the value a given number of seconds ahead. It is commonly used to alert before a disk or quota runs out.

What are DORA's five software delivery metrics?

Change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. The last two measure instability.

What is deployment rework rate?

The share of deployments that are unplanned and triggered by a production incident. A rising value means more release effort goes into fixing production problems.

How many failure-prediction alerts should a service have?

Only as many as people will act on. Start with three to six leading indicators matched to how the service fails, page only for imminent problems and send slow trends to tickets.

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.