Top 20 CI/CD Failures Observed in Production

20 CI/CD failures that pass a green pipeline and break production, from global config pushes to one-way migrations, with fixes and public postmortems.

Dec 22, 2025 - 18:09
Nov 7, 2026 - 23:52
 0  69
Top 20 CI/CD Failures Observed in Production

The CI/CD failures that hurt production are the ones the pipeline reports as green: a config file pushed everywhere at once, a code path with no feature flag, a migration that cannot be rolled back, or a health check that passes while users see errors. Fixing them means staged rollouts, kill switches, compatible changes and real verification.

This list is about what breaks after the deploy succeeds. Job-level problems such as flaky tests, expired runner tokens and full disks are covered in 12 common CI/CD pipeline failures and fixes, and pipeline design mistakes are in 10 real-world CI/CD design errors. Both sit under our CI/CD pipeline examples hub. Where a failure has a well-known public postmortem, it is linked so you can read the provider's own account.

Key takeaways

  • Most large outages in public postmortems began with a change that passed every check and was then sent to all servers at once.
  • Treat configuration and data pushes like code: version them, validate them, and roll them out in stages.
  • Every risky change needs a switch that turns it off faster than a redeploy.
  • During a rolling deploy two versions run side by side, so APIs, events and database schemas must work with both.
  • A rollback plan that has never been rehearsed is a guess. So is a backup that has never been restored.
  • Post-deploy checks must measure what users feel (errors, latency), not whether a process is up.

The 20 failures at a glance

#FailureStage where it startsFirst defence
1Global push with no staged rolloutReleaseRings or waves with bake time
2New code path without a feature flagBuildFlag off by default, kill switch
3Generated config trusted blindlyReleaseValidate size and schema at load
4Bad data replicated everywhere in secondsRuntimeIncremental propagation with checks
5Operational tool with no guardrailsOperationsRate limits and minimum capacity checks
6Restart storm after recoveryRecoveryJittered exponential backoff
7Locking or breaking schema migrationDeployExpand and contract migrations
8Incompatible API change during rolloutDeployBackward-compatible contracts
9Shallow health checkVerifyChecks that exercise dependencies
10Traffic before the app is readyDeployReadiness probes and warm-up
11One-way change blocks rollbackDeployRehearsed rollback, roll-forward plan
12Canary nobody is watchingVerifyAutomated canary analysis
13Expired certificate or rotated secretRuntimeExpiry monitoring, staged rotation
14Config that exists in staging onlyReleaseConfig in Git, validated per environment
15Production runs a different artifactReleaseImmutable tags and digests
16Compromised dependency shippedBuildPinning, SBOM, provenance checks
17Resource limits tuned for stagingDeployLoad tests and right-sized limits
18Front end and back end out of stepReleaseVersioned assets, ordered deploys
19Backup that cannot be restoredOperationsScheduled restore tests
20Alerts that never reach anyoneMonitorTest the alert path end to end

Release and rollout failures

1. A global push with no staged rollout

The change goes to every host or customer at the same moment, so a defect becomes a full outage before anyone can react. CrowdStrike's root cause analysis of the July 2024 Channel File 291 incident lists the lack of staged deployment for that content type among its findings and commits to staged deployment of such updates. Fix: ship in rings (internal, 1%, 10%, 50%, 100%) with a bake time and automatic halt on error signals. Rollout types are compared in 12 CI/CD deployment rollout types.

2. A new code path without a feature flag

New logic is deployed live and only runs when a rare input arrives, so the rollout looks healthy for days. Google's report on the 12 June 2025 Google Cloud outage says a new quota policy check had been rolled out region by region, but the failing path only ran when a specific policy change arrived, and the change "did not have appropriate error handling nor was it feature flag protected." Fix: wrap new paths in flags that default to off, enable them in stages, and keep a kill switch that operators can flip without a deploy.

3. Generated configuration trusted as safe

Teams validate user input carefully but assume files produced by their own systems are fine. Cloudflare's write-up of its 18 November 2025 outage describes a database permissions change that made a query emit duplicate rows, doubling the size of a generated Bot Management feature file. The file was pushed across the network and exceeded a limit in the proxy, which then failed with 5xx errors. Cloudflare said it would harden ingestion of internally generated configuration in the same way as user input. Fix: validate size, schema and value ranges when config is loaded, and keep the last good version to fall back to.

4. Bad data replicated everywhere in seconds

Global replication is great for latency and bad for blast radius. In the Google Cloud incident, the policy data with blank fields replicated globally within seconds, so every region hit the same crash. Fix: propagate data changes incrementally, check health between steps, and alert on unusual change volume.

5. An operational tool with no guardrails

Not every production change goes through the pipeline. AWS's summary of the February 2017 S3 disruption in us-east-1 explains that a mistyped input to an established playbook command removed far more servers than intended, and that "the tool used allowed too much capacity to be removed too quickly." AWS changed the tool to remove capacity more slowly and to block removals below a minimum. Fix: put rate limits, dry-run modes and safety floors in every runbook script, and route routine operations through the same review as code.

6. A restart storm after recovery

Once the fix is out, thousands of tasks restart together and overload a shared dependency. Google's report notes that in its largest region, restarting tasks overloaded the underlying database because the service lacked randomized exponential backoff. Fix: add jitter and backoff to clients and start-up code, and throttle task creation during recovery.

Deploy-time failures

7. A schema migration that locks or breaks

A migration adds a column with a default to a huge table, renames a field the old code still reads, or holds a lock long enough to stall every request. Fix: use expand and contract. First add new structures in a backward-compatible way, deploy code that writes both, backfill in batches, switch reads, and remove the old structure in a later release. Run migrations as a separate, observable step rather than inside application start-up.

Release 1: add column email_normalized (nullable)  -- old and new code both work
Release 2: app writes email and email_normalized; batch backfill
Release 3: app reads email_normalized
Release 4: drop old column after a full release cycle

8. An incompatible API change during a rolling update

During a rolling deploy, version N and N+1 serve traffic together, and queues can hold messages written by either. A renamed JSON field or a new required parameter breaks whichever side did not expect it. Fix: add fields before you require them, accept old and new formats for at least one release, and run contract tests between services in CI.

9. A health check that passes while the app is broken

An endpoint returns 200 because the web server is up, even though the database connection pool is exhausted. Load balancers keep sending traffic and the deploy is marked successful. Fix: keep liveness simple, but make readiness and post-deploy checks exercise the critical path (a real query, a dependency call with a short timeout). The checks to run after release are listed in 12 post-deployment validation steps.

10. Traffic before the app is ready

New pods receive requests while caches are cold, JIT compilation is running or configuration is still loading, so the first minutes after every deploy show a latency spike or errors. Fix: define readiness probes, use minReadySeconds on Kubernetes Deployments, add warm-up steps, and make sure old instances drain connections before they stop.

spec:
  minReadySeconds: 20
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 25%
  template:
    spec:
      containers:
        - name: api
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            failureThreshold: 3

11. A one-way change that blocks rollback

The release dropped a column, changed a data format or upgraded a client library that rewrote stored state. Rolling the code back now makes things worse. Fix: mark irreversible changes in the pull request, split them from code changes, rehearse rollback in staging, and decide in advance when you will roll forward instead.

12. A canary that nobody is watching

A canary only helps if something compares it with the baseline and stops the rollout. Many teams send 5% of traffic to the new version and then promote on a timer. Fix: define pass criteria (error rate, p99 latency, saturation) and let the deployment tool promote or roll back automatically. Argo Rollouts, Flagger and AWS CodeDeploy with CloudWatch alarms all support this. Gate ideas are in 10 CI/CD quality gates for production reliability.

Configuration, artifact and supply chain failures

13. An expired certificate or a rotated secret

The code did not change, but a TLS certificate expired or a password was rotated in the vault while running pods still held the old value. Fix: monitor certificate expiry with alerts weeks ahead, automate renewal, and rotate secrets in two steps (add new, deploy, remove old) so both values work during the switch.

14. Configuration that exists only in staging

A new environment variable or feature setting was added by hand in staging and forgotten in production. The pipeline passes; the app crashes on start or falls back to an unsafe default. Fix: keep configuration in Git per environment, have the application fail fast on missing required settings, and diff environments as a pipeline step.

15. Production runs a different artifact than the one tested

The image is rebuilt for each stage, or a mutable tag such as latest moves between test and release, so what reached production was never tested. Fix: build once, promote the same artifact, and deploy by digest.

image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/api@sha256:3f1c...e9a2

16. A compromised dependency shipped to production

A malicious or hijacked package version enters through a floating version range. The xz Utils backdoor tracked as CVE-2024-3094 showed that even core open-source libraries can carry malicious code in released versions. Fix: pin versions with lock files, generate an SBOM, scan in CI, verify signatures or provenance where available, and use a private proxy registry. Security controls are covered in 14 CI/CD security best practices.

17. Resource limits tuned for staging

Memory limits that were fine with staging data cause OOM kills under production load, or CPU limits throttle the service at peak. Fix: load test with production-like data, set requests from observed usage, alert on throttling and restarts, and roll limit changes out like any other change.

18. Front end and back end out of step

New HTML references JavaScript bundles that a CDN has not yet received, or a mobile client calls an endpoint that has not been deployed. Fix: publish versioned static assets before the HTML that references them, keep old assets available for a while, and deploy back ends before the clients that depend on them.

Operations and monitoring failures

19. A backup that cannot be restored

The pipeline takes a "pre-deploy backup" and nobody checks it. GitLab's postmortem of its January 2017 database incident explains that after data was deleted by mistake, the regular pg_dump backups turned out to be failing because of a version mismatch, and the failure emails had been rejected. Fix: restore backups on a schedule into a scratch environment and alert if the restore fails or is late.

20. Alerts that never reach anyone

The monitoring caught the problem, but the alert went to an old channel, an expired on-call rota or a mailbox that filtered it. Fix: send a synthetic test alert through the full path every week, and review routing whenever teams change. The GitLab postmortem above is a reminder that silent failure notifications are part of the problem.

A pre-release checklist built from these failures

  1. Is the change behind a flag, and can an operator turn it off in under five minutes?
  2. Will it roll out in stages, with automatic halt on error rate or latency?
  3. Can version N and N+1 run together against the same database and queues?
  4. If the change touches data or schema, is there a tested rollback or a written roll-forward plan?
  5. Is the artifact the same digest that passed tests?
  6. Are all required settings present in production, and does the app fail fast if one is missing?
  7. Do post-deploy checks exercise the real user path?
  8. Has the alert route been tested recently?

Practise handling production failures

These failures are easier to recognise once you have caused and fixed a few in a lab. The DevOps training in Pune covers pipelines, rollout strategies, monitoring and incident response with hands-on labs, in the Kharadi classroom or live online. For Jenkins-based pipelines, see the Jenkins training.

Frequently asked questions

Why do production failures happen when the pipeline passed?

Pipelines test code in controlled conditions. Production adds real data, real traffic, mixed versions during rollout and changes outside the code such as configuration, certificates and replicated data. The fixes are staged rollouts, flags and checks that watch real user signals after release.

What is the most common cause of large outages in public postmortems?

A change, often configuration or data rather than code, reaching every server at once without staged rollout or a fast way to turn it off. The CrowdStrike, Cloudflare and Google Cloud reports linked in this article each describe a version of this.

How is this list different from CI/CD pipeline failures?

Pipeline failures stop the pipeline: a red build, a failed test, an expired token. The failures here get through a green pipeline and break production. See our separate guide on common pipeline failures for job-level fixes.

What is an expand and contract migration?

It is a way to change a database schema without downtime. You first add the new structure alongside the old one, update code to use both, backfill data, switch reads to the new structure and only remove the old one in a later release.

Should every deployment use a canary?

Not always. Canaries help for services with enough traffic to compare error rates. Low-traffic services may do better with blue/green plus synthetic tests. Whatever you choose, promotion should depend on measured signals, not a timer.

How often should we test backups and rollback?

Restore backups on a schedule, for example weekly for critical data, and alert if a restore fails. Rehearse rollback for every change that touches data, and run a full game day a few times a year.

What is a kill switch in deployments?

A setting, usually a feature flag, that disables a feature or code path immediately without a new deploy. It shortens an incident from the length of a release cycle to the time it takes to flip a flag.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.