Why Latency Monitoring Matters in Microservices
Why tail latency grows with fan-out in high-volume microservices, how to measure p99 with Prometheus histograms, and how to alert on latency SLOs.
Latency monitoring matters in high-volume microservices because one user request fans out to many services, and the slowest call sets the response time. Averages hide that slow tail. Measure latency as histograms, alert on high percentiles such as p99 tied to an SLO, and use distributed tracing to find which hop is slow.
This post explains why tail latency grows with request volume and fan-out, how to measure it correctly with Prometheus histograms, and how to turn the numbers into alerts people trust. It sits in our observability cluster; for the full tool list see top observability tools for DevOps monitoring.
Key takeaways
- Latency is one of Google's four golden signals, next to traffic, errors and saturation.
- In a fan-out architecture, a rare slow call in one backend becomes a common slow response at the front. The Google SRE book notes that one backend's 99th percentile can become the frontend's median.
- Track latency as a distribution (histogram buckets), not as an average.
- Percentiles cannot be averaged across instances. Aggregate histogram buckets first, then compute the quantile.
- Measure the latency of failed requests separately; a slow error is worse than a fast one.
- Alert on SLO burn, not on every percentile wobble. Use traces and exemplars to get from "p99 is high" to "this call is slow".
Why volume and fan-out make latency worse
A monolith serving a page might make one database call. A microservice page might call an API gateway, an auth service, a catalogue service, a pricing service and a recommendations service, each of which calls its own dependencies. The page is only as fast as the slowest path through that graph.
Jeff Dean and Luiz Andre Barroso described this in "The Tail at Scale" (Communications of the ACM, 2013): when a request touches many servers in parallel, even a small fraction of slow responses on each server makes slow overall responses common. The arithmetic is simple. If each backend call is fast 99 percent of the time and a request waits on 100 such calls, the chance that all 100 are fast is 0.99 raised to the power 100, about 37 percent. Most requests hit at least one slow call.
High volume adds pressure in other ways:
- Queueing. As utilisation rises, waiting time in thread pools, connection pools and message queues grows faster than load.
- Retries. A slow dependency triggers client retries, which add load and can turn slowness into an outage.
- Shared resources. Garbage collection pauses, noisy neighbours, and CPU throttling under container limits all show up as tail latency first.
Latency usually moves before errors do, so it is often your earliest warning of saturation.
What to measure
The Google SRE book chapter on monitoring distributed systems defines latency as the time it takes to service a request, and lists it as the first of the four golden signals. Practical measurement points:
| Where | What it shows | Typical source |
|---|---|---|
| Edge (load balancer, API gateway) | What users experience end to end | Gateway or ingress metrics, access logs |
| Service server side | Time each service spends handling a request | Instrumented HTTP or gRPC middleware |
| Client side of each call | Server time plus network, DNS, TLS and queueing | Client libraries, service mesh proxies |
| Dependencies | Database, cache and queue latency | Exporters, cloud provider metrics |
| Real users | Browser or app experience | Real user monitoring |
Split every latency metric by route or method and by outcome (success or error). The SRE book warns that fast HTTP 500 errors can pull an average down and make a failing service look healthy, so track error latency rather than filtering it out.
Percentiles, explained without maths jargon
| Statistic | Meaning | Use it for |
|---|---|---|
| Average | Total time divided by count | Capacity estimates; not for user experience |
| p50 (median) | Half of requests are faster than this | Typical experience |
| p95 | 95 percent of requests are faster | Most users' worst normal case |
| p99 | 99 percent of requests are faster | The tail that fan-out amplifies |
| p99.9 | 999 in 1000 requests are faster | Very high-volume or critical paths |
| Max | Slowest single request | Spotting outliers; too noisy to alert on |
At high volume, p99 is not rare. A service handling 2,000 requests per second sends 20 requests per second slower than its p99, which is 72,000 requests an hour.
Measuring correctly with Prometheus histograms
Prometheus offers histograms and summaries. A summary computes quantiles inside each process, and the Prometheus histogram guide states plainly that you cannot aggregate those quantiles. Averaging p95 values from ten pods gives a meaningless number. A histogram exposes bucket counts, which you can add up across pods and then turn into a quantile on the server.
Classic histogram query for p95 across all replicas of a service:
histogram_quantile(
0.95,
sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))
)
Keep the route label if you want a percentile per endpoint:
histogram_quantile(
0.99,
sum by (le, route) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))
)
Prometheus also supports native histograms, which the docs recommend where available because they have finer resolution without hand-picked buckets. For a native histogram the query drops the le grouping: histogram_quantile(0.95, sum(rate(http_request_duration_seconds[5m]))). Client library support is still limited, so check yours before switching.
Choosing buckets
- Put bucket boundaries around your SLO threshold. If the target is 300 ms, have buckets at, for example, 0.25 and 0.3 and 0.5 seconds so the quantile is accurate where it matters.
- The SRE book suggests roughly exponential spacing for latency buckets so a histogram covers a wide range.
- Each bucket is a separate time series per label combination. Avoid high-cardinality labels such as user ID or full URL.
From percentiles to SLOs and alerts
A latency SLO states what fraction of requests must be faster than a threshold over a window, for example "99 percent of checkout requests complete in under 300 ms over 30 days". With histograms you can compute the good-request ratio directly:
sum(rate(http_request_duration_seconds_bucket{service="checkout", le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{service="checkout"}[5m]))
Alert when the error budget burns too fast, using a short and a long window together so brief spikes do not page anyone but sustained slowness does. The SRE workbook chapter on alerting on SLOs walks through multi-window, multi-burn-rate alerts. Keep raw p99 graphs on dashboards for investigation rather than paging on them directly.
groups:
- name: checkout-latency
rules:
- alert: CheckoutLatencyBudgetBurn
expr: |
(
1 - (
sum(rate(http_request_duration_seconds_bucket{service="checkout",le="0.3"}[1h]))
/
sum(rate(http_request_duration_seconds_count{service="checkout"}[1h]))
)
) > (14.4 * 0.01)
and
(
1 - (
sum(rate(http_request_duration_seconds_bucket{service="checkout",le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{service="checkout"}[5m]))
)
) > (14.4 * 0.01)
labels:
severity: page
annotations:
summary: "Checkout is burning its latency error budget fast"
The factor 14.4 is the fast-burn rate the SRE workbook uses as an example, checked over one hour and five minutes together; at that rate a 30 day budget lasts about two days. The 0.01 is the error budget of a 99 percent SLO. Adjust both to your own target.
Finding the slow hop: tracing and exemplars
Metrics tell you that p99 rose. Traces tell you where the time went. A distributed trace records each call in a request as a span with start time and duration, so a waterfall view shows which service or query added the delay. OpenTelemetry is the CNCF project that standardises how services emit traces, metrics and logs; most tracing backends (Jaeger, Grafana Tempo, Zipkin and commercial tools) accept its data.
Exemplars link the two. A histogram bucket can carry the trace ID of a sample request, so from a latency spike on a Grafana panel you can jump straight to a trace that fell into the slow bucket. Our posts on distributed tracing tools and auto-instrumentation for tracing cover setup.
At high volume you cannot keep every trace. Head sampling (decide at the start) is cheap; tail sampling (decide after the trace completes) lets you keep all slow or failed traces and a small share of normal ones, which is what you want for latency work.
Common causes of tail latency and what to check
| Cause | Signal to look for | Typical fix |
|---|---|---|
| Slow database queries | Long DB spans, rising DB latency metrics | Indexes, query rewrites, caching, read replicas |
| Connection or thread pool exhaustion | Time spent waiting before the call starts | Right-size pools, add timeouts, shed load |
| CPU throttling in containers | Container CPU throttling metrics climbing | Review CPU limits and requests |
| Garbage collection pauses | Latency spikes matching GC logs | Heap tuning, fewer allocations |
| Retry storms | Traffic to a dependency rises while it slows | Retry budgets, backoff with jitter, circuit breakers |
| Cross-zone or DNS delays | Long client-side time, short server-side time | Topology-aware routing, DNS caching |
| Cold starts and scaling lag | Spikes after deploys or scale-out | Readiness probes, warm-up, pre-scaling |
"The Tail at Scale" also describes techniques such as hedged requests (send a backup request if the first is slow) for read paths where duplicate work is acceptable. Use them carefully, because they add load.
A rollout plan for a team starting out
- Instrument every service with request duration histograms, split by route and outcome. OpenTelemetry or your framework's Prometheus middleware can do this.
- Build one dashboard per service with p50, p95 and p99, request rate and error rate.
- Pick one or two user-facing journeys and write latency SLOs for them.
- Add burn-rate alerts for those SLOs and retire noisy threshold alerts.
- Add tracing with tail sampling for slow and failed requests, and enable exemplars.
- Review latency after every incident and every major release.
For metrics that pair well with latency, see 18 observability metrics for cloud-native systems, and for the reasoning behind moving past simple dashboards, observability vs traditional monitoring. To decide which signal helps fastest during an incident, read which observability pillar gives the fastest incident insight.
Our Prometheus and Grafana training covers histograms, PromQL and alerting rules hands on, in our Kharadi, Pune classroom and live online.
Frequently asked questions
Why is latency monitoring important in microservices?
One user request calls many services, so a slow call anywhere slows the whole response. Latency monitoring shows where time is spent and often warns of saturation before errors appear.
What is p99 latency?
p99 is the response time that 99 percent of requests beat. The slowest 1 percent are above it. At high request rates that 1 percent is a large number of real users.
Why not use average latency?
Averages hide the slow tail and can be pulled down by fast errors. Percentiles from histograms show what slower requests actually experience.
Can I average p95 values from several pods?
No. Prometheus documentation states that quantiles cannot be aggregated. Sum histogram bucket rates across pods first, then apply histogram_quantile.
Should I alert directly on p99 latency?
Usually not. Define a latency SLO and alert on how fast its error budget burns, using short and long windows. Keep p99 graphs for investigation.
How does distributed tracing help with latency?
A trace records each call in a request with its duration, so you can see which service, query or network hop added the delay. Exemplars link latency metrics to sample traces.
What is tail latency amplification?
When a request waits on many parallel backend calls, a small chance of slowness in each call becomes a large chance that at least one is slow, so the overall response is slow far more often.
Which tools are used for latency monitoring?
Common choices are Prometheus and Grafana for metrics, OpenTelemetry for instrumentation, and Jaeger, Grafana Tempo or Zipkin for traces, plus commercial APM tools.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0