12 Post-Deployment Validation Steps in CI/CD

Post-deployment validation proves a release works in production before you call it done. The 12 steps are: confirm the rollout finished, check the running version, check health, run smoke tests, run synthetic journeys, compare golden signals, gate promotion on analysis, scan logs for new errors, check drift, validate data, run a passive security scan, then record the result and decide.

A green pipeline only shows that the build passed its tests and the deploy command exited cleanly. It does not show that users can log in, that the new pods talk to the right database, or that error rates stayed flat. This guide turns those questions into 12 checks you can automate, with commands and configuration checked against the Kubernetes, Argo CD, Argo Rollouts, Flagger, Helm, Prometheus, Grafana, Terraform and OWASP ZAP documentation. It is part of our CI/CD series that starts with CI/CD pipeline examples from big tech companies. The checks that run before a release are covered in CI/CD quality gates; this post picks up the moment the deploy starts.

Key takeaways

• Treat validation as part of the deployment job. If a check fails, the pipeline should fail and roll back, not send an email someone reads tomorrow.
• Run fast, shallow checks first (rollout status, version, health, smoke tests) and slower, statistical checks after (metric comparison, canary analysis).
• Compare against a baseline, not a fixed number. An error rate of 0.4% means little unless you know it was 0.1% before the deploy.
• Synthetic tests catch problems at night and on low-traffic paths where real user metrics are too thin to show a regression.
• Passive security scans, such as the OWASP ZAP baseline scan, are safe to run against production because they do not send attacks.
• Write every deployment as an annotation on your dashboards and record the outcome. That is how you measure change fail rate.

The 12 validation steps at a glance

Step 1: Confirm the rollout actually finished

Many pipelines run kubectl apply and exit successfully while new pods are still crash-looping. Make the job wait for the rollout and fail if it does not complete:

kubectl apply -f k8s/
kubectl rollout status deployment/web -n shop --timeout=5m

kubectl rollout status returns a non-zero exit code if the rollout fails or the timeout passes. Kubernetes also marks a stuck Deployment with a Progressing condition set to false and reason ProgressDeadlineExceeded after progressDeadlineSeconds (600 seconds by default). For GitOps deployments, wait on Argo CD instead:

argocd app wait shop-web --sync --health --timeout 600

Step 2: Verify the version that is running

Confirm production runs the artifact you built, not a cached tag. Deploy images by digest and compare the digest in the cluster with the one your build produced, and expose a small version endpoint that returns the commit SHA:

EXPECTED_SHA="$GITHUB_SHA"
LIVE_SHA=$(curl --fail --silent --max-time 5 https://shop.example.com/version | jq -r .commit)
if [ "$LIVE_SHA" != "$EXPECTED_SHA" ]; then
  echo "Expected $EXPECTED_SHA but production reports $LIVE_SHA"
  exit 1
fi
kubectl get deployment web -n shop -o jsonpath='{.spec.template.spec.containers[0].image}'

Reusing a mutable tag like latest is a common reason this check fails. Behind a CDN or load balancer, also check that old instances are no longer serving traffic.

Step 3: Health and dependency checks

Kubernetes probes are the first line of validation. A readiness probe keeps a pod out of the Service until it can serve, a liveness probe restarts a hung container, and a startup probe gives slow-starting apps time before liveness checks begin. Point readiness at an endpoint that checks what the pod needs to serve requests, such as its database connection, but keep liveness simple so a database outage does not restart every pod.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 5

Rolling updates only move on when new pods are ready, so a good readiness endpoint stops a bad release from replacing healthy pods.

Step 4: Smoke tests on critical paths

Smoke tests answer one question: is the release broken on arrival? Keep them to a handful of requests that cover the paths that make money or block users: the home page, login, search, checkout API, and one call through each critical dependency. They should finish in under a few minutes and fail loudly.

#!/usr/bin/env bash
set -euo pipefail
BASE="https://shop.example.com"
for path in / /api/health /api/products?limit=1 /login; do
  code=$(curl --silent --output /dev/null --write-out '%{http_code}' --max-time 10 "$BASE$path")
  echo "$path -> $code"
  [ "$code" -lt 400 ] || { echo "Smoke test failed on $path"; exit 1; }
done

Two Kubernetes-native ways to run them:

• Helm chart tests: a pod template with the annotation "helm.sh/hook": test, run with helm test <release>.
• Argo CD resource hooks: a Job annotated argocd.argoproj.io/hook: PostSync. Argo CD runs PostSync hooks after the sync succeeds and all resources are healthy.

apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: curlimages/curl:8.10.1
          args: ["--fail", "--max-time", "10", "http://web.shop.svc.cluster.local/api/health"]

Pair the PostSync hook with a SyncFail hook or an alert so a failed smoke test pages someone or triggers a rollback.

Step 5: Synthetic user journeys

Synthetic monitoring runs scripted requests or browser sessions against production on a schedule, so you find out a flow is broken even when no real user is trying it. Right after a deploy, trigger the same suite once and wait for the result.

For simple HTTP and TLS checks, the Prometheus blackbox_exporter probes a URL through its /probe endpoint and exposes a probe_success metric you can alert on:

scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://shop.example.com/login
          - https://shop.example.com/api/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

For multi-step journeys such as sign up, add to cart and pay with a test card, use browser automation (Playwright, Selenium) or a managed synthetic monitoring service from your cloud or observability vendor. Use dedicated test accounts and mark synthetic orders so they never reach fulfilment or analytics.

Step 6: Compare golden signals against the baseline

Google's SRE book names four golden signals for user-facing systems: latency, traffic, errors and saturation. After a deploy, compare each with the same window before the deploy, or with the previous version if both are running. A query for the share of 5xx responses over the last five minutes:

sum(rate(http_requests_total{job="web",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="web"}[5m]))

And the 95th percentile latency from a histogram:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="web"}[5m])))

Add one or two business signals that the deploy could affect, such as successful logins per minute or orders per minute. Airbnb wrote about adding business metrics to its deployment checks after finding that some costly bugs did not change system metrics at all. Our guide to deployment observability signals lists more signals worth tracking.

Step 7: Gate promotion on automated canary analysis

If you roll out progressively, let the controller run the comparison in step 6 and decide. In Argo Rollouts, an AnalysisTemplate queries Prometheus and fails the rollout if the success rate drops:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 5m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(irate(istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}",response_code!~"5.*"}[5m]))
            /
            sum(irate(istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}"}[5m]))

Flagger does the same through its analysis block and adds webhooks at fixed points: pre-rollout runs before traffic reaches the canary, rollout runs on every analysis iteration (a good place for load or acceptance tests), confirm-promotion holds promotion until a check returns HTTP 200, and post-rollout runs after promotion or rollback. The rollout types themselves are explained in CI/CD deployment rollout types.

Step 8: Scan logs and error tracking for new errors

Metrics show that something changed; logs and error trackers show what. Tag every log line and error event with the release version or commit SHA, then look for error types that never appeared before this release. A new exception group seen only on the new version is a strong rollback signal even when the overall error rate looks normal.

Useful checks in this window:

• Count of ERROR and FATAL log lines per minute, split by version.
• New error groups created since the deploy started.
• Container restarts: kubectl get pods -n shop and the RESTARTS column, plus kubectl logs <pod> --previous for a crashed container.
• Warnings you added on purpose for this release, such as a fallback path being used.

Step 9: Check configuration and infrastructure drift

Validate that production matches what is in Git. In Argo CD, the application should be Synced and Healthy; an OutOfSync status right after a deploy means something in the cluster differs from the repository. For infrastructure managed by Terraform, a scheduled plan with -detailed-exitcode exits with code 2 when the real infrastructure no longer matches the configuration:

terraform plan -detailed-exitcode -input=false -lock=false
# 0 = no changes, 1 = error, 2 = changes (drift) present

Also confirm that environment-specific values (feature flag defaults, connection strings, replica counts) are the production values and not staging leftovers.

Step 10: Validate data and migrations

Database changes are the hardest part of a deploy to undo, so check them explicitly:

• The migration job completed and the schema version table shows the expected version.
• New columns and indexes exist; long-running index builds have finished.
• Row counts or checksums for tables touched by a data migration match expectations.
• The previous application version can still read and write the new schema, so a rollback does not break.

Use the expand and contract pattern: add new structures first, deploy code that works with both old and new, and remove old columns in a later release only after the new version is stable.

Step 11: Run a passive security check

A release can weaken security without breaking any function, for example by dropping a security header, exposing a debug endpoint or serving an expired certificate. Run checks that only observe:

• The OWASP ZAP baseline scan spiders the target for about a minute by default and runs passive checks only, without attacks. It exits with 1 on a FAIL and 2 on WARN only, so a pipeline can decide what blocks.
• Check that the TLS certificate is valid and not close to expiry, and that HTTP redirects to HTTPS.
• Confirm admin and debug paths return 401, 403 or 404 from the public internet.

docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://shop.example.com

Run active scans only against staging or a dedicated test environment, never against production users. Our guide to OWASP ZAP in CI/CD covers both modes.

Step 12: Record the deployment and decide

End every deploy with a decision: promote, hold or roll back. Make it visible by writing a deployment marker to your dashboards so anyone looking at a graph can see when the change happened. Grafana's HTTP API accepts annotations at POST /api/annotations:

curl --fail -X POST "https://grafana.example.com/api/annotations" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"time\": $(date +%s000), \"tags\": [\"deploy\", \"web\"], \"text\": \"web $GITHUB_SHA deployed\"}"

Grafana's docs note that the /api routes are being superseded by /apis routes starting in Grafana 13, but the legacy endpoints remain available. Record whether the deploy needed a rollback or hotfix. Over time that gives you DORA's change fail rate and failed deployment recovery time. If you need to undo a release, our post on deployment rollback strategies compares the options.

Putting it together in a pipeline

A GitHub Actions deploy job that runs the fast checks in order and fails the workflow if any fails:

deploy:
  runs-on: ubuntu-latest
  environment: production
  steps:
    - uses: actions/checkout@v7
    - name: Deploy
      run: kubectl apply -f k8s/
    - name: Wait for rollout
      run: kubectl rollout status deployment/web -n shop --timeout=5m
    - name: Check version
      run: ./scripts/check-version.sh "$GITHUB_SHA"
    - name: Smoke tests
      run: ./scripts/smoke.sh
    - name: Passive security scan
      run: docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://shop.example.com
    - name: Roll back on failure
      if: failure()
      run: kubectl rollout undo deployment/web -n shop

Steps 6 to 8 need a bake period, so they usually run in the rollout controller (Argo Rollouts or Flagger) or in a later pipeline stage that waits 15 to 30 minutes and queries your monitoring system. Pick the wait time based on how long your traffic takes to exercise the change, not a fixed rule.

Common mistakes

Learn to build validated deployments

Our DevOps training in Pune walks through pipelines that deploy, test and roll back on their own, in classroom sessions in Kharadi and live online across India. To go deeper on the monitoring side of steps 5 to 8, see the Prometheus and Grafana training.

Frequently asked questions

What is post-deployment validation in CI/CD?

It is the set of automated checks that run after code reaches an environment, usually production, to confirm the release works: the rollout completed, the right version is live, health checks and smoke tests pass, and error rates and latency have not regressed compared with before the deploy.

What is the difference between smoke testing and post-deployment validation?

Smoke testing is one step within validation. It sends a few requests to critical paths to see whether the release is broken on arrival. Full validation also compares metrics with a baseline, checks logs for new errors, verifies data and configuration, and ends with a promote or rollback decision.

How long should post-deployment validation take?

Fast checks (rollout status, version, health, smoke tests) should finish within a few minutes. Metric comparison and canary analysis need enough traffic to be meaningful, often 15 to 30 minutes per step for busy services and longer for quiet ones.

How do I run smoke tests after an Argo CD sync?

Create a Kubernetes Job with the annotation argocd.argoproj.io/hook: PostSync. Argo CD runs it after the sync succeeds and all resources are healthy. Add hook-delete-policy: HookSucceeded so successful jobs are cleaned up, and alert or roll back if the job fails.

Should validation failures trigger automatic rollback?

For clear failures such as a failed rollout, failed smoke test or a canary analysis failure, yes. Argo Rollouts, Flagger and AWS CodeDeploy can all roll back automatically. For ambiguous signals, alert a person and hold promotion instead of rolling back.

Is it safe to run security scans against production?

Passive checks are. The OWASP ZAP baseline scan spiders for about a minute and only observes responses, without attacking. Run active scans, which send attack payloads, against staging or a dedicated test environment.

Which metrics should I compare after a deployment?

Start with the four golden signals from Google's SRE book: latency, traffic, errors and saturation. Add one or two business metrics the change could affect, such as logins or orders per minute, and compare each with the period before the deploy.

How does post-deployment validation relate to DORA metrics?

Validation decides whether a deployment needs immediate intervention. Recording that outcome gives you DORA's change fail rate, and timing the recovery gives failed deployment recovery time. Good validation catches failures sooner, which shortens recovery.

References

• Kubernetes: Deployments and rollout status
• Kubernetes: Configure liveness, readiness and startup probes
• Argo CD: Sync phases, waves and resource hooks
• Argo CD: argocd app wait
• Argo Rollouts: Prometheus analysis
• Flagger: Webhooks
• Helm: Chart tests
• Prometheus blackbox_exporter
• Google SRE book: Monitoring distributed systems (four golden signals)
• Grafana: Annotations HTTP API
• Terraform: plan command
• OWASP ZAP: Baseline scan
• DORA: Software delivery performance metrics
• Airbnb Tech Blog: How Airbnb Safeguards Changes in Production