9 Real DevOps Automation Failures and What They Taught
Nine public DevOps automation failures from SEC, AWS, Google, Cloudflare, GitLab and others, with sources, causes and the controls that would have helped.
The best-documented DevOps automation failures share a few causes: a change reached everything at once, a safety check failed or did not exist, old code or data was still in play, or nobody had tested recovery. Below are nine incidents from public postmortems and regulator filings, each with the source, what happened and the control that would have limited it.
Every incident here comes from a primary source that you can read yourself: an SEC order, or an engineering postmortem published by the affected company. We quote no numbers that those sources do not give, and we avoid guessing at causes they do not state. The goal is to learn from automation that went wrong, not to blame the people involved. For broader pipeline problems read Top 20 CI/CD Failures Observed in Production and Why Do DevOps Failures Happen.
Key takeaways
- Automation makes a mistake fast and wide. A single incorrect input, rule or flag can reach every server or region within seconds.
- Staged rollout is the most repeated lesson: Cloudflare (2019), CrowdStrike (2024) and Google Cloud (2025) all changed their processes to roll out changes gradually.
- Safety checks need their own tests. Facebook's audit tool had a bug that let a maintenance command through, and Atlassian's deletion API accepted the wrong kind of ID without warning.
- Recovery that has never been tested is not recovery. GitLab's backups failed when they were needed, and Atlassian missed its restore-time target for many sites at once.
- Automation can contain race conditions too. AWS's DynamoDB DNS automation, in October 2025, deleted records through a race between two automated components.
The nine failures at a glance
| # | Incident | Date | Failure pattern | Main control that was missing or weak |
|---|---|---|---|---|
| 1 | Knight Capital | 1 Aug 2012 | Partial deployment, old code reactivated | Deployment verification and review |
| 2 | GitLab.com database | 31 Jan 2017 | Wrong host, backups that did not work | Tested restores and backup alerts |
| 3 | Amazon S3 | 28 Feb 2017 | Wrong input to a capacity-removal tool | Limits on how much one command can remove |
| 4 | Cloudflare WAF | 2 Jul 2019 | Global rule push, no staged rollout | Progressive rollout and CPU protection |
| 5 | Facebook backbone | 4 Oct 2021 | Audit tool bug let a command run | A working pre-flight check |
| 6 | Atlassian site deletions | 5 Apr 2022 | Script given the wrong kind of ID | Soft deletes and API validation |
| 7 | CrowdStrike content update | Jul 2024 | Faulty content, no staged deployment | Validation and staged deployment |
| 8 | Google Cloud Service Control | 12 Jun 2025 | Bad data replicated globally | Feature flag and error handling |
| 9 | AWS DynamoDB DNS | 19-20 Oct 2025 | Race condition in DNS automation | Freshness checks and safe cleanup |
1. Knight Capital: one server missed the deployment (2012)
Source: the SEC's order in the matter of Knight Capital Americas LLC (Release No. 34-70694, October 2013).
What happened: Knight was rolling out new code for a retail liquidity program to its order-routing system, which ran on eight servers. The new code was meant to replace unused code called Power Peg, which Knight had stopped using years earlier but which was still present and callable. The new code also reused a flag that had once switched Power Peg on. A technician did not copy the new code to one of the eight servers, no second technician reviewed the deployment, and Knight had no written procedure requiring one. On 1 August 2012, orders carrying the reused flag reached the eighth server and triggered the old code. In about 45 minutes the system sent millions of child orders, which resulted in 4 million executions in 154 stocks. The SEC states Knight lost more than $460 million.
The lesson: verify that every target actually received the release, review manual deployment steps, and do not reuse an old flag for a new meaning. Delete dead code instead of leaving it callable. Immutable images and a pipeline that deploys to all hosts from one artifact remove the "forgot one server" class of error. Our guide to deployment rollback strategies covers going back quickly when a release misbehaves.
2. GitLab.com: a recovery nobody had tested (2017)
Source: GitLab's postmortem, "Postmortem of database outage of January 31."
What happened: While rebuilding a database replica after a load spike, an engineer ran a data-directory wipe on the primary database instead of the secondary. Recovery then showed that the regular backup had uploaded nothing: it ran from an application server whose PostgreSQL client tools were a different version from the database, and the failure emails were rejected because of email authentication settings. Disk snapshots were not enabled for the database servers, and the replica had been wiped as part of the rebuild. GitLab restored from a manual snapshot taken about six hours earlier. Database changes made between 17:20 and 23:30 UTC on 31 January were lost, including at least 5,000 projects, 5,000 comments and roughly 700 user accounts. Git repositories and wikis were stored separately and were not affected.
The lesson: a backup you have not restored is a hope. Alert on backup failure through a channel that works, make the host you are on obvious in the shell, and give backup health an owner.
3. Amazon S3: a typo with a large radius (2017)
Source: AWS, "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region."
What happened: At 9:37 AM PST on 28 February 2017, an authorized team member ran a command, following an established playbook, to remove a small number of servers from an S3 billing subsystem. One of the inputs was entered incorrectly, so a much larger set of servers was removed. They supported the index and placement subsystems, which both had to restart fully. S3 and the services that depend on it in the region were affected for about 4 hours and 17 minutes, until 1:54 PM PST.
The lesson: AWS changed the tool to remove capacity more slowly and added safeguards that block removals that would drop a subsystem below its minimum capacity. It also began auditing other tools for similar checks. Rate limits and minimums inside the tool itself work better than relying on the operator to type correctly.
4. Cloudflare: a rule pushed everywhere at once (2019)
Source: Cloudflare, "Details of the Cloudflare outage on July 2, 2019."
What happened: A new web application firewall rule containing a regular expression that backtracked heavily was deployed globally. The rule was distributed within seconds and exhausted CPU on the processing cores that serve HTTP and HTTPS traffic worldwide. Contributing factors were a CPU-protection safeguard that had been removed by mistake in a refactor, a regex engine without complexity guarantees, a test suite that did not check for excessive CPU use, and an operating procedure that allowed a non-emergency rule change to skip a staged rollout. The rule went live at 13:42 UTC and traffic recovered by 14:09, after the WAF was disabled globally at 14:07. Cloudflare reported 27 minutes of impact.
The lesson: Cloudflare restored the CPU protection, checked all 3,868 WAF managed rules, added performance profiling to the tests, planned a move to a regex engine with run-time guarantees, and changed the procedure so rule changes roll out in stages, while keeping an emergency path for active attacks. Configuration is code and needs the same staged release. See 12 Kubernetes Rollout Strategies with Examples for gradual release patterns.
5. Facebook: the safety check had a bug (2021)
Source: Facebook Engineering, "More details about the October 4 outage."
What happened: During routine maintenance, an engineer issued a command intended to assess the availability of global backbone capacity. It unintentionally took down all connections in the backbone network, disconnecting data centers worldwide. Facebook's systems normally audit commands like this to prevent mistakes, but a bug in the audit tool prevented it from properly stopping the command.
The lesson: a guardrail needs its own tests. If a check is the only barrier between a command and a global outage, test it with commands that should be refused. Facebook's post says the company would strengthen testing and drills, and look for ways to simulate a global backbone outage in future exercises.
6. Atlassian: a script deleted whole sites (2022)
Source: Atlassian, post-incident review of the April 2022 outage.
What happened: On 5 April 2022, Atlassian ran an existing script to remove a legacy app from customer sites. The script was meant to receive app IDs, but it was given site IDs, and the deletion API accepted either kind without warning. The result was 883 sites deleted, affecting 775 customers, between 07:38 and 08:01 UTC. The peer review checked which endpoint was called, not whether the IDs matched the app, and staging did not reveal the problem because the IDs did not exist there. Some customers were offline for up to 14 days; all sites were restored by 18 April. Atlassian reported that it met its data-loss target (no more than about five minutes of data lost, with a small number of exceptions) and missed its restore-time target.
The lesson: Atlassian blocked bulk site deletions, required soft deletes with a retention period, added staged rollouts and tested rollback plans for deletion operations, and accelerated its work on restoring many sites at once. Destructive APIs should validate the type of what they are given, and destructive scripts should run on a few items first.
7. CrowdStrike: content shipped without stages (2024)
Source: CrowdStrike, root cause analysis of the Channel File 291 incident (August 2024).
What happened: A content update to the Falcon sensor in July 2024 caused Windows hosts to crash. The analysis lists these findings: the number of fields in the template type was not validated when the sensor was compiled, the content interpreter had no runtime bounds check on the input array, the content validator contained a logic error, and testing of the template type did not cover a wide enough range of matching criteria. It also notes that template instances had no staged deployment process.
The lesson: the planned mitigations include runtime bounds checks, wider testing, a fixed validator, validation of template instances inside the interpreter, and staged deployment for template instances. Data and content that ships on its own schedule needs the same canary and staged release as code.
8. Google Cloud: bad data replicated everywhere (2025)
Source: Google Cloud status page, incident report for 12 June 2025.
What happened: A policy change with unintended blank fields was inserted into the regional database tables used by Service Control. Because quota metadata replicates globally within seconds, the bad data reached every region and triggered a null pointer bug in a code path added on 29 May. The code had no appropriate error handling and was not protected by a feature flag, so the binaries crashed in a loop. The "red button" kill switch was deployed about 40 minutes into the incident and started recovery. The status dashboard lists the incident from 10:51 PDT to 18:18 PDT, with the largest region taking longest to recover because restarts overloaded the underlying table.
The lesson: Google committed to making failing checks fail open so API requests are still served, to propagating globally replicated data in increments with time to validate it, to requiring feature flags that are disabled by default for changes to critical binaries, and to improving error-handling tests and exponential backoff. Anything replicated globally deserves a delay and a validation step.
9. AWS DynamoDB: automation raced itself (2025)
Source: AWS, summary of the Amazon DynamoDB service disruption in the Northern Virginia (US-EAST-1) Region, October 2025.
What happened: DynamoDB's DNS management has two parts: a DNS Planner that creates plans of load balancers and weights, and DNS Enactors that apply them to Route 53 in three Availability Zones. One Enactor was unusually delayed while retrying updates. Meanwhile the Planner produced newer plans, and another Enactor applied one and cleaned up older plans. The delayed Enactor then applied its much older plan over the newer one, and its freshness check was stale by then. The cleanup then deleted that old plan, which removed all IP addresses for the regional endpoint and left the system in an inconsistent state that needed manual operator intervention. The event ran from 11:48 PM PDT on 19 October to 2:20 PM PDT on 20 October, and it affected EC2 instance launches, network load balancer health checks, Lambda and other services.
The lesson: AWS disabled the Planner and Enactor automation worldwide, planned to fix the race condition and add protections against incorrect plans before re-enabling it, and added a velocity control that limits how much capacity a single load balancer can remove. Checks done at the start of a long operation must be repeated before the final write, and cleanup jobs should never be able to delete the only live copy.
Patterns across the nine failures
| Pattern | Seen in | Control to build |
|---|---|---|
| Change reaches everything at once | Cloudflare, CrowdStrike, Google Cloud | Staged rollout with automatic halt |
| Wrong input to a powerful tool | Amazon S3, Atlassian, GitLab | Input validation, minimum-capacity limits, soft deletes |
| Safety check missing or broken | Facebook, Knight Capital, Atlassian | Test the check; require second-person review |
| Untested recovery | GitLab, Atlassian | Restore drills with recorded timings |
| Hidden old code or state | Knight Capital, Google Cloud | Delete dead code; flag new paths |
| Race or stale check in automation | AWS DynamoDB | Re-validate before write; safe cleanup |
A checklist you can apply this week
- List your five most powerful automated actions (delete, scale down, rotate, push config) and write down what limits each one.
- For each, add a maximum blast radius per run, such as a cap on instances removed, and a dry-run mode.
- Put configuration and data changes through the same staged rollout as code, and halt automatically on a rising error rate.
- Run one restore drill and record how long it takes.
- Test a safety check by sending it a command it should refuse.
- Write a kill switch for each new feature on a critical path, and practise using it.
These habits are practised in labs in our DevOps training in Pune and DevSecOps training in Pune, both available as live online batches across India. For the self-healing side of this topic, see 10 Ways to Implement Self-Healing in DevOps Systems and how chaos engineering improves resilience.
Frequently asked questions
What is the most common cause of DevOps automation failure?
In these public postmortems the most repeated cause is a change that reached too many systems at once without a staged rollout or an automatic stop. The second is a powerful tool that accepted a wrong input.
How much did the Knight Capital deployment failure cost?
The SEC order states Knight lost more than $460 million from the unwanted positions, in about 45 minutes of trading on 1 August 2012.
Why did GitLab lose data in the 2017 incident?
An engineer wiped the wrong database host while rebuilding a replica, and the regular backups, snapshots and replica were not usable for a fast recovery. Changes made between 17:20 and 23:30 UTC on 31 January were lost.
Can automation be safe in production?
Yes, when each automated action has a limit on its scope, a dry run, a staged rollout, a way to stop it and a tested recovery path. The incidents above are mostly cases where one of those was missing.
What is a staged rollout?
It is releasing a change to a small portion of systems first, checking health metrics, and widening only if they stay healthy. Cloudflare and CrowdStrike both moved their processes toward staged rollouts after their incidents.
Should every action be automated?
No. Irreversible actions such as deleting data or removing capacity deserve extra limits and often a human approval. Automate the routine path and keep a person in the loop for the risky one.
Where can I read these postmortems myself?
Each section names its source, and the links are in the references below. Reading a postmortem in full is one of the fastest ways to learn how real systems fail.
Official references
- SEC: Knight Capital Americas LLC, Release No. 34-70694
- GitLab: postmortem of database outage of January 31
- AWS: summary of the Amazon S3 service disruption, February 2017
- Cloudflare: details of the outage on July 2, 2019
- Facebook Engineering: more details about the October 4 outage
- Atlassian: post-incident review of the April 2022 outage
- CrowdStrike: Channel File 291 incident root cause analysis
- Google Cloud: incident report, 12 June 2025
- AWS: summary of the Amazon DynamoDB service disruption, October 2025
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0