14 CI/CD Pipeline Patterns for Faster Results
14 CI/CD pipeline patterns that cut build and delivery time: path filters, caching, test sharding, merge queues, preview environments, GitOps and canaries.
The fastest CI/CD pipelines share a set of patterns: run only what changed, cache dependencies and build layers, parallelise and shard tests, fail fast, build once and promote, use merge queues, preview environments, GitOps delivery and progressive rollouts. Combine the ones that remove your slowest stage first, then measure again.
Pipeline speed decides how often developers get feedback and how quickly a fix reaches users. A pipeline that takes an hour invites people to batch changes and ignore results. This guide describes fourteen pipeline patterns that cut feedback and delivery time, each with what it speeds up, a short configuration example from GitHub Actions or GitLab CI/CD, and the trade-off to watch. Every syntax example was checked against the vendor's documentation. For real company examples, see our CI/CD pipeline examples from big tech companies, the hub of this series.
Key takeaways
- Measure stage durations before optimising. The slowest stage, not the most visible one, deserves attention first.
- Skipping unchanged work (path filters, affected-only builds) often saves more than making work faster.
- Caching and parallel test shards are the two most common wins for build and test time.
- Order stages so cheap checks fail first, and cancel superseded runs.
- Delivery speed comes from building one artifact, promoting it, and deploying with GitOps and progressive rollouts.
- Faster is only useful if the pipeline stays reliable. Flaky tests erase speed gains.
The fourteen patterns at a glance
| # | Pattern | Speeds up | Main trade-off |
|---|---|---|---|
| 1 | Path-filtered pipelines | Whole runs | Missed dependencies between folders |
| 2 | Affected-only monorepo builds | Build and test | Needs a build graph tool |
| 3 | Dependency caching | Install steps | Stale or poisoned caches |
| 4 | Container layer caching | Image builds | Cache storage and invalidation |
| 5 | Parallel jobs (DAG) | Pipeline wall time | More runner minutes |
| 6 | Test sharding | Test stage | Uneven shards |
| 7 | Fail-fast ordering | Feedback on failures | Some later signals arrive later |
| 8 | Cancel superseded runs | Queue time | Lost results for older commits |
| 9 | Right-sized runners | CPU-bound steps | Cost per minute |
| 10 | Merge queue | Main branch stability and merge flow | Extra CI runs per merge |
| 11 | Build once, promote | Release stages | Requires externalised config |
| 12 | Ephemeral preview environments | Review and QA | Cluster or cloud cost |
| 13 | GitOps delivery | Deploy and promotion | Repository design effort |
| 14 | Progressive delivery with auto-rollback | Safe production rollout | Needs good metrics |
1. Path-filtered pipelines
Do not run the backend test suite when someone edits a README. Both major CI services let a workflow or job run only when certain paths change.
# GitHub Actions
on:
pull_request:
paths:
- 'services/api/**'
- '.github/workflows/api.yml'
# GitLab CI/CD
api-test:
script: make -C services/api test
rules:
- changes:
- services/api/**/*
Watch for shared code. If services/api imports from libs/common, include that path too, or required checks may be skipped when they should run.
2. Affected-only builds in monorepos
Path filters get unwieldy in large monorepos. Build tools that understand the dependency graph compute which projects a change affects and run only those targets. Nx has nx affected, Turborepo has filters based on changed packages, and Bazel builds only invalidated targets. Combined with a remote cache, unchanged targets are fetched rather than rebuilt.
npx nx affected -t lint test build --base=origin/main --head=HEAD
The trade-off is adopting and maintaining the build tool. It pays off once a repository holds many projects.
3. Dependency caching
Downloading npm, Maven, pip or Go modules on every run wastes minutes. Cache them keyed on the lock file so the cache refreshes only when dependencies change.
# GitHub Actions
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
# GitLab CI/CD
test:
cache:
key:
files:
- package-lock.json
paths:
- .npm/
script:
- npm ci --cache .npm --prefer-offline
- npm test
Scope caches per branch where security matters; a cache written by an untrusted fork should not feed protected-branch builds.
4. Container layer caching
Docker images rebuild fast when layers are ordered well (dependencies before source code) and when the build reuses a cache between runners. BuildKit can export cache to a registry or, on GitHub Actions, to the Actions cache.
- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v6
with:
push: true
tags: ghcr.io/acme/api:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
More techniques, such as multi-stage builds and cache mounts, are in 12 ways to speed up Docker builds in CI/CD.
5. Parallel jobs as a dependency graph
Strict stages make every job wait for the slowest job in the previous stage. Define dependencies explicitly so each job starts as soon as its inputs are ready. GitLab uses needs; GitHub Actions jobs run in parallel by default and use needs to order them.
# GitLab CI/CD
build-api:
stage: build
script: make build-api
test-api:
stage: test
needs: [build-api]
script: make test-api
lint:
stage: test
needs: []
script: make lint
needs: [] lets lint start immediately instead of waiting for the build stage.
6. Test sharding
Split a long test suite across several runners. GitHub Actions uses a matrix; GitLab offers the parallel keyword with CI_NODE_INDEX and CI_NODE_TOTAL variables. Many test runners accept a shard argument directly.
# GitHub Actions with Playwright
strategy:
matrix:
shard: [1, 2, 3, 4]
steps:
- run: npx playwright test --shard=${{ matrix.shard }}/4
# GitLab CI/CD
e2e:
parallel: 4
script:
- npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
Shards finish only as fast as the slowest one. Balance them by historical timing where your runner supports it.
7. Fail-fast stage ordering
Put the cheapest, most likely-to-fail checks first: formatting, lint, type checks and unit tests. Expensive integration and end-to-end tests run only after those pass. A broken build then reports in two minutes instead of twenty. In a matrix, GitHub Actions cancels remaining matrix jobs when one fails, controlled by strategy.fail-fast (true by default). The quality checks themselves are covered in CI/CD quality gates for production-level reliability.
8. Cancel superseded runs
When a developer pushes three commits in a row, only the last one matters. Cancelling older runs frees runners and shortens queues.
# GitHub Actions
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
# GitLab CI/CD
workflow:
auto_cancel:
on_new_commit: interruptible
test:
interruptible: true
script: make test
Do not mark deployment jobs as interruptible; cancelling a deploy halfway can leave an environment in a mixed state.
9. Right-sized and self-hosted runners
Compilation and heavy test suites are often CPU-bound. GitHub offers larger hosted runners and GitLab offers larger SaaS runner sizes; both also support self-hosted runners, which can keep warm caches on local disk. On Kubernetes, Actions Runner Controller and the GitLab Runner Kubernetes executor scale runners with demand. Compare cost per minute against minutes saved, and remember self-hosted runners need patching and isolation.
10. Merge queue
When many developers merge into one branch, a change can pass CI on its own branch and still break main when combined with another change. Teams then re-run pipelines, revert, and lose time. A merge queue tests each change on top of the changes ahead of it and merges only when that combined build passes. GitHub offers merge queues on protected branches (workflows listen for the merge_group event), and GitLab has merge trains.
on:
pull_request:
merge_group:
Main stays green, which keeps continuous deployment from main safe.
11. Build once, promote the same artifact
Rebuilding per environment repeats work and means production runs something you did not test. Build one image tagged with the commit SHA, push it, and promote it by changing which tag each environment uses. Config lives outside the image in ConfigMaps, secrets or Helm values. Promotion then takes seconds. Our guide to reducing deployment time with DevOps covers this and the process side of release speed.
12. Ephemeral preview environments
A shared staging environment becomes a queue. Creating a short-lived environment per pull request lets reviewers and testers see the change immediately and removes waiting for a slot. GitLab's review apps use environment with on_stop; on Kubernetes, the Argo CD ApplicationSet pull request generator creates an application per open pull request.
# GitLab CI/CD
deploy-review:
stage: deploy
script: ./deploy.sh review-$CI_MERGE_REQUEST_IID
environment:
name: review/$CI_COMMIT_REF_SLUG
on_stop: stop-review
auto_stop_in: 2 days
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
Set auto_stop_in or a TTL so forgotten environments do not pile up costs.
13. GitOps delivery
Instead of a pipeline pushing to clusters with stored credentials, a controller in each cluster (Argo CD or Flux) pulls the desired state from Git and applies it. CI builds and pushes the image, then updates the tag in a config repository. Deployments become small commits, multiple clusters update in parallel, and drift is corrected automatically. See GitOps deployment best practices and CI/CD solutions for Kubernetes workloads for setups.
14. Progressive delivery with automatic rollback
The last step of a fast pipeline is a safe production rollout that needs no one watching. Canary analysis shifts traffic in steps and checks metrics between them; if error rates rise, it rolls back on its own. Argo Rollouts expresses this declaratively:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api
spec:
strategy:
canary:
steps:
- setWeight: 10
- pause: {duration: 5m}
- setWeight: 50
- pause: {duration: 5m}
Add an AnalysisTemplate that queries Prometheus to abort automatically when a success-rate threshold fails. Flagger offers similar automation with service meshes and ingress controllers.
Which patterns to apply first
| Symptom | Start with |
|---|---|
| Every change runs everything | 1, 2 |
| Install and image steps dominate | 3, 4 |
| Tests take most of the time | 5, 6, 7 |
| Long queues for runners | 8, 9 |
| Main branch often broken | 10 |
| Releases wait for staging or approvals | 11, 12 |
| Deploys are manual or risky | 13, 14 |
Track pipeline duration, queue time and failure rate per stage after each change; our list of CI/CD monitoring tools helps here. To build these pipelines with guidance, see our GitLab CI/CD training or the broader DevOps training, in Kharadi, Pune or live online.
Frequently asked questions
What makes a CI/CD pipeline fast?
Doing less work (path filters, affected-only builds, caching), doing work in parallel (job graphs, test shards), failing early, and removing waits in delivery with promotion, preview environments and automated rollouts.
How do I find the slowest part of my pipeline?
Look at stage and job durations plus queue time across recent runs in your CI tool's UI or API, or use a pipeline analytics tool. Optimise the largest share of total time first.
Is caching safe in CI pipelines?
Yes, if caches are keyed on lock files and scoped so untrusted branches cannot write caches that protected branches read. Treat caches as disposable and rebuildable.
What is a merge queue?
A feature that tests each pull request together with the changes queued ahead of it and merges only if the combined result passes. GitHub calls it a merge queue; GitLab calls it a merge train.
Does GitOps make deployments faster?
It makes promotion a small Git commit and lets clusters sync in parallel without pipeline credentials. The speed gain comes mostly from simpler, repeatable releases rather than raw deploy time.
How does test sharding work?
The test suite is split into N parts that run on N runners at once. Each runner gets its shard number, for example via a GitHub Actions matrix or GitLab's parallel keyword, and runs only its share.
Should every team use all fourteen patterns?
No. Small projects may need only caching and fail-fast ordering. Pick patterns that address your measured bottleneck, and add more as the codebase and team grow.
References
- GitHub Actions workflow syntax
- GitHub documentation: Managing a merge queue
- GitLab CI/CD YAML syntax reference
- GitLab documentation: Merge trains
- Docker documentation: Cache storage backends
- Nx documentation: Run only tasks affected by a PR
- Argo CD ApplicationSet pull request generator
- Argo Rollouts documentation
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0