12 Deployment Observability Signals to Track
12 signals that show whether a deploy is healthy: markers, rollout stalls, errors and p99 by version, restarts, saturation, traces and DORA metrics.
Deployment observability signals tell you, within minutes of a release, whether the new version is healthy and whether it is worse than the one it replaced. The twelve signals below cover the rollout itself, the users' experience, the platform, and the delivery process. Each one is tagged by version so you can compare old and new side by side.
General observability metrics are covered in 18 observability metrics for cloud-native systems, and dashboard layouts in 12 monitoring dashboards every DevOps should build. This post is narrower: the signals that answer "did this deploy go well?" For the wider tool picture, see the observability tools pillar.
Key takeaways
- A deployment signal is only useful if you can split it by version. Tag telemetry with
service.versionand mark each deploy on your dashboards. - Watch four groups: rollout mechanics (did it finish?), user impact (errors, latency, conversions), platform health (restarts, saturation) and delivery outcomes (DORA metrics).
- Kubernetes marks a stalled rollout with
reason: ProgressDeadlineExceededafterprogressDeadlineSeconds(default 600), andkubectl rollout statusexits non-zero when that happens. - DORA's current five metrics are change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate.
- Compare new against old, not against a fixed threshold alone. A 0.5% error rate may be normal for one service and a disaster for another.
The 12 signals at a glance
| # | Signal | Group | Answers |
|---|---|---|---|
| 1 | Deployment markers | Context | When did the change happen? |
| 2 | Rollout progress and stall condition | Rollout | Did the new version finish rolling out? |
| 3 | Ready vs desired replicas | Rollout | Are new pods actually serving? |
| 4 | Error rate by version | User impact | Is the new version failing more requests? |
| 5 | Tail latency by version | User impact | Is it slower for the slowest users? |
| 6 | Business conversion rate | User impact | Can users still complete key actions? |
| 7 | Client-side errors and crashes | User impact | Is the front end or mobile app breaking? |
| 8 | Container restarts and termination reasons | Platform | Is the new version crashing or being OOM-killed? |
| 9 | Resource saturation per pod | Platform | Does the new version use more CPU or memory? |
| 10 | New log error signatures | Platform | Are there errors we have never seen before? |
| 11 | Dependency error rate from traces | Platform | Is the new version hurting downstream services? |
| 12 | DORA delivery metrics | Delivery | Over many deploys, are we getting better? |
Before you start: tag everything with a version
Every signal below depends on separating old from new. In OpenTelemetry, set the resource attributes service.name, service.version and deployment.environment.name. The semantic conventions mark deployment.environment.name as stable.
export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES="service.version=${GIT_SHA},deployment.environment.name=production"
In Kubernetes, also put the version in a pod label (for example app.kubernetes.io/version) so metrics scraped by Prometheus carry it. With Argo Rollouts, canary and stable pods get different rollouts-pod-template-hash labels you can use the same way.
Context signal
1. Deployment markers
A vertical line on every dashboard at the moment of a deploy is the cheapest observability win there is. When a graph changes shape, the first question is "what changed?", and a marker answers it. In Grafana, your pipeline can create an annotation through the annotations HTTP API:
curl -s -X POST "$GRAFANA_URL/api/annotations" \
-H "Authorization: Bearer $GRAFANA_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"tags\":[\"deploy\",\"checkout\"],\"text\":\"checkout ${GIT_SHA} by ${CI_PIPELINE_ID}\"}"
Leaving out dashboardUID creates an organisation annotation that any dashboard can show through a tag query. Include the commit SHA and pipeline link in the text so on-call engineers can jump straight to the change.
Rollout signals
2. Rollout progress and the stall condition
A Deployment that never finishes is a failed deploy, even if no error appears. Kubernetes tracks this with .spec.progressDeadlineSeconds, which defaults to 600 seconds. When the deadline passes, the Deployment gets a Progressing condition with status: "False" and reason: ProgressDeadlineExceeded. Kubernetes takes no other action, so your pipeline must react.
if ! kubectl rollout status deployment/checkout -n shop --timeout=10m; then
echo "Rollout failed or stalled, rolling back"
kubectl rollout undo deployment/checkout -n shop
exit 1
fi
kubectl rollout status returns a non-zero exit code when the progress deadline is exceeded, which makes it a reliable pipeline gate.
3. Ready vs desired replicas
Track the gap between desired and available replicas during and after the rollout. kube-state-metrics exposes kube_deployment_spec_replicas and kube_deployment_status_replicas_available. A gap that persists after the rollout ends often means pods are failing readiness probes intermittently.
kube_deployment_spec_replicas{deployment="checkout"}
- kube_deployment_status_replicas_available{deployment="checkout"}
User impact signals
4. Error rate by version
The share of requests returning server errors, split by version, is the first signal most canary analysis uses. Compare the canary with the stable version over the same window:
sum by (version) (rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
/
sum by (version) (rate(http_requests_total{service="checkout"}[5m]))
Metric and label names depend on your instrumentation; the pattern is what matters. Alert on the difference between versions, not just the absolute value.
5. Tail latency by version
Averages hide the slowest requests. Use p95 or p99 from histograms:
histogram_quantile(0.99,
sum by (le, version) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m])))
A new version that adds a database call per request often shows up here before it shows up in errors.
6. Business conversion rate
Some defects return HTTP 200 and still break the product: a button that does nothing, a price that renders as zero. Track one or two business events per service, such as checkouts completed per minute or sign-ups per hundred visits, and plot them with deploy markers. If you run canaries, emit these as metrics labelled by version.
7. Client-side errors and crashes
Front-end JavaScript errors and mobile crashes never reach your server logs. Real user monitoring and crash reporting tools report them per release version. Watch crash-free sessions after a mobile release and JavaScript error rate after a web release, and link each to the release that introduced it.
Platform signals
8. Container restarts and termination reasons
A new version that crashes on a code path, or leaks memory, shows up as restarts. kube-state-metrics exposes kube_pod_container_status_restarts_total (stable) and kube_pod_container_status_last_terminated_reason (experimental), which carries reasons such as OOMKilled.
increase(kube_pod_container_status_restarts_total{namespace="shop",container="checkout"}[15m]) > 0
Also watch kube_pod_container_status_waiting_reason for CrashLoopBackOff and ImagePullBackOff right after a deploy.
9. Resource saturation per pod
Compare CPU and memory per pod between versions. A release that uses 30% more memory may be fine today and cause OOM kills at peak traffic next week. cAdvisor metrics such as container_memory_working_set_bytes and container_cpu_usage_seconds_total, grouped by pod label, make the comparison possible. Check throttling too if you set CPU limits.
10. New log error signatures
Counting error lines is noisy, because every service has background errors. A better signal is errors whose message pattern did not exist before the deploy. Most log platforms (Loki, Elasticsearch, cloud logging services) can group logs by pattern; compare the top patterns in the 30 minutes after a deploy with the hour before.
11. Dependency error rate from traces
A new version can be healthy itself and still hurt others, for example by sending more queries to a shared database or calling an API with a bad payload. Distributed traces show error rate and latency on each outgoing call, broken down by the caller's service.version. Check the services your release calls, not only the one you deployed.
Delivery signal
12. DORA delivery metrics
Signals 1 to 11 judge a single deploy. DORA metrics judge your process across many. The current DORA guide groups five metrics:
| Group | Metric | Definition |
|---|---|---|
| Throughput | Change lead time | Time from commit to running in production |
| Throughput | Deployment frequency | How often you deploy |
| Throughput | Failed deployment recovery time | Time to recover from a deploy that needs immediate intervention |
| Instability | Change fail rate | Share of deploys needing immediate intervention, such as a rollback or hotfix |
| Instability | Deployment rework rate | Share of deploys that are unplanned and caused by a production incident |
Feed them from your pipeline and incident tool: each deploy event, each rollback, each hotfix. Signals 2, 4 and 8 are what tell you a deploy needed intervention, so the per-deploy signals and the DORA metrics share the same data.
Turning signals into automatic decisions
Signals help most when they decide something without a person staring at a graph:
- Pipeline gate:
kubectl rollout statuswith a timeout, followed by a short smoke test. - Canary analysis: Argo Rollouts or Flagger querying signals 4 and 5 by version, aborting automatically. See why progressive rollouts matter for customer-facing apps.
- Post-deploy watch: an alert rule that is only active for 30 to 60 minutes after a deploy marker, with tighter thresholds than normal.
- Weekly review: DORA metrics and the list of deploys that failed, discussed by the team.
Which signals matter for which kind of change
Not every deploy needs every signal with the same weight. Match the watch list to what changed:
| Type of change | Signals to weight most | Why |
|---|---|---|
| Application code | 4, 5, 6, 10 | Logic bugs show up as errors, slowdowns, broken flows and new log messages |
| Dependency or base image upgrade | 8, 9, 10 | Library and runtime changes often affect memory, startup and crash behaviour |
| Front-end bundle | 6, 7 | Server metrics stay flat while browsers throw errors |
| Configuration or feature flag | 1, 4, 6, 11 | No new pods roll out, so rollout signals tell you nothing; markers become essential |
| Database migration | 5, 11 | Locks and missing indexes appear as latency and dependency errors |
| Infrastructure (node pool, ingress, mesh) | 2, 3, 5, 8 | Scheduling, readiness and network path changes hit rollout and latency first |
Configuration changes deserve special attention. They skip the rollout machinery entirely, so a flag flip or a ConfigMap update can change behaviour for every user at once. Record them as deploy markers too, with the same tags you use for code releases.
Alert noise during deploys
Rollouts cause short, expected blips: pods restart, caches are cold, connection pools refill. If your normal alerts fire on these, people learn to ignore deploy-time alerts, which is the opposite of what you want. Two practices help. First, use rates over a window of several minutes rather than instant values, so a single restart does not page anyone. Second, prefer the comparison between versions over absolute thresholds during the rollout; a cold-cache latency spike that affects both versions equally is not a release problem. Keep a separate, stricter alert for signals that should never move during a deploy, such as checkout failures.
A post-deploy checklist
- Deploy marker created with SHA and pipeline link.
- Rollout finished before the progress deadline.
- Available replicas equal desired replicas.
- Error rate and p99 latency for the new version within agreed limits of the old version.
- Key business metric steady.
- No new restarts, OOM kills or CrashLoopBackOff.
- No new top log error patterns.
- Downstream services show no new errors.
- Deploy recorded for DORA, with outcome.
Common mistakes
- No version label. Without it, a canary's errors are averaged into the stable version and disappear.
- Watching only infrastructure. Healthy pods can serve a broken product. Include at least one business signal.
- Fixed thresholds for every service. Use the stable version as the baseline.
- Too short a watch window. Memory leaks and cache warm-up effects take time. Keep watching after the rollout reaches 100%.
- Treating DORA as a target. DORA metrics describe outcomes; gaming deployment frequency with empty deploys teaches you nothing.
Learn to build these signals
For the monitoring stack itself, read how to use Prometheus and Grafana together and the difference between observability and traditional monitoring. The Prometheus and Grafana training covers PromQL, alerting and dashboards hands-on, in the Kharadi, Pune classroom or live online.
Frequently asked questions
What are deployment observability signals?
Metrics, logs, traces and events that show whether a specific release is healthy, ideally split by version so the new release can be compared with the one it replaced.
What is the most important signal after a deploy?
Error rate by version, followed closely by tail latency. Together they catch most user-visible regressions, but add a business metric to catch silent failures.
How do I know if a Kubernetes rollout has stalled?
After progressDeadlineSeconds (600 by default) the Deployment's Progressing condition becomes False with reason ProgressDeadlineExceeded, and kubectl rollout status exits with a non-zero code.
What are the DORA metrics in 2027?
The DORA guide lists five: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate, grouped into throughput and instability.
How do I add deploy markers to Grafana?
Have the pipeline POST to Grafana's /api/annotations endpoint with tags and a text field containing the version and pipeline link, then show those tags on your dashboards.
Why compare the new version with the old one instead of using fixed thresholds?
Each service has its own normal error and latency levels. Comparing versions over the same time window removes traffic and time-of-day effects and shows what the release changed.
How long should I watch a deploy?
Watch closely during the rollout and for at least 30 to 60 minutes after it reaches full traffic. Keep a longer view for memory and saturation, which can drift over hours.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0