10 Best Practices for Scaling Jenkins Agents

Scale Jenkins agents with the Kubernetes and EC2 Fleet plugins: ephemeral agents, labels, caps, spot capacity, caching, WebSocket and queue metrics.

Dec 24, 2025 - 18:13
Nov 8, 2026 - 07:06
 0  124
10 Best Practices for Scaling Jenkins Agents

To scale Jenkins agents, stop adding static machines and let a cloud plugin create agents on demand: the Kubernetes plugin for pods, or the EC2 and EC2 Fleet plugins for virtual machines. Give each agent one job, label agents by capability, cap capacity, cache dependencies outside the workspace, and watch the build queue to size it.

This post is about the agent layer only: how many agents you run, where they come from and how they connect. For pipeline style, security and plugin hygiene, see our 20 Jenkins best practices, and for the plugins themselves see 12 Jenkins plugins that make CI/CD easier. The parent guide for this topic is our overview of CI/CD tools.

Key takeaways

  • The controller schedules work; agents run it. Run zero executors on the built-in node so builds never compete with the controller for CPU and memory.
  • Ephemeral agents (one build per agent) remove leftover state and make capacity elastic.
  • The Kubernetes plugin creates a pod per build from a pod template. The EC2 Fleet plugin scales an Auto Scaling group or Spot Fleet.
  • Labels route builds to the right machine type. Keep the label set small and descriptive.
  • Build queue length and wait time are the numbers that tell you whether you have enough agents.

How Jenkins distributes builds

A Jenkins controller holds configuration, the job queue and build records. Agents connect to it and offer executors, which are slots that run one build step each. When a pipeline reaches a node or agent block, Jenkins finds an idle executor whose agent matches the requested label. If none exists, the build waits in the queue. Cloud plugins watch that queue and start new agents when work is waiting.

Agent typeHow it startsGood forMain drawback
Static (SSH or inbound)Always runningSpecial hardware, licensed tools, macOS buildsIdle cost, state drift between builds
Kubernetes podKubernetes plugin creates a pod per buildContainer builds, many small jobsDocker-in-Docker and privileged builds need care
Cloud VMEC2, EC2 Fleet, Azure VM Agents or Google Compute Engine pluginHeavy builds, full OS accessSlower start than pods
Docker container on a hostDocker plugin starts a container per buildSmall teams with one or two build hostsCapacity limited to the hosts you own

The 10 practices

1. Take builds off the controller

Set the number of executors on the built-in node to 0 (Manage Jenkins, Nodes, Built-In Node). Jenkins documentation recommends this for security as well as performance: a build running on the controller can read files in JENKINS_HOME, including secrets. With zero executors, every build must go to an agent, which is what makes the rest of this list work.

2. Make agents ephemeral

An ephemeral agent runs one build and is then removed. There is no leftover workspace, no cached credentials from another job, and no "works on agent 3 but not agent 5" problem. Both the Kubernetes plugin and the EC2 plugins can terminate agents after use. For VM agents, the EC2 Fleet plugin has a "max total uses" setting; set it to 1 for single-use machines, or a small number if start-up time is high.

3. Run agents as pods with the Kubernetes plugin

The Kubernetes plugin starts a pod for each build. The pod always contains a jnlp container running the jenkins/inbound-agent image, which connects back to the controller. You add tool containers next to it and choose which one runs each step.

pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
          - name: maven
            image: maven:3.9-eclipse-temurin-21
            command: ["sleep"]
            args: ["infinity"]
            resources:
              requests:
                cpu: "1"
                memory: 2Gi
              limits:
                memory: 2Gi
      '''
      defaultContainer 'maven'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh 'mvn -B package'
      }
    }
  }
}

Keep pod templates in a shared library or in the cloud configuration managed by Configuration as Code, so teams do not copy YAML into every Jenkinsfile.

4. Set resource requests on every agent pod

Without requests, the Kubernetes scheduler packs agent pods onto nodes until builds run out of memory or fight for CPU. Set CPU and memory requests that match the real build, and a memory limit to stop one build from taking a node down. If you run the Cluster Autoscaler or Karpenter, requests are also what trigger new nodes when pending pods cannot fit. Our guide to Kubernetes scaling strategies explains how node autoscaling responds to pending pods.

5. Use spot or preemptible capacity for interruptible builds

Most CI builds can be retried, which makes them a good fit for spot instances. On AWS, the EC2 Fleet plugin can scale an Auto Scaling group with a mixed instances policy or a Spot Fleet. On Kubernetes, put a spot node pool behind a taint and add a matching toleration to the build pod template. Keep release and deployment jobs on on-demand capacity, and add retry around stages that can be interrupted.

6. Label agents by capability, not by name

Labels such as linux && docker, arm64 or gpu describe what an agent can do. Pipelines ask for capabilities; Jenkins picks any matching agent. Avoid labels tied to one machine, such as build-server-02, because they turn a pool into a set of single points of failure.

pipeline {
  agent { label 'linux && arm64' }
  stages {
    stage('Test') { steps { sh 'make test' } }
  }
}

7. Cap capacity and set idle timeouts

Elastic does not mean unlimited. In the Kubernetes cloud settings, the concurrency limit caps how many agent pods can run; each pod template can have its own instance cap. The EC2 plugins have instance caps and idle termination times. Caps protect your budget and your downstream systems (artifact repositories and test databases can fall over before Jenkins does). Idle timeouts remove agents that are not needed, while a short idle period lets a burst of builds reuse warm agents.

8. Cache dependencies outside the workspace

Ephemeral agents start empty, so every build downloads Maven, npm or pip dependencies again unless you plan for it. Options, from simplest to most involved:

  • Run a repository proxy such as Nexus or Artifactory close to the agents.
  • Bake common tools and dependencies into the agent image, and rebuild it on a schedule.
  • Mount a shared cache volume (for example a PersistentVolumeClaim with ReadWriteMany) for package caches, accepting that concurrent writes need care.

Our Docker image optimization guide covers building small, cache-friendly agent images.

9. Connect agents with WebSocket where networking is awkward

Inbound agents can connect over a dedicated TCP port or over WebSocket through the controller's normal HTTP port. WebSocket support (added in Jenkins 2.217) avoids opening an extra port through load balancers and ingress controllers. On Kubernetes, the plugin has a WebSocket option in the cloud configuration. Agents across a slow or distant network link also benefit from keeping workspaces small, because remoting transfers files through the controller connection.

10. Measure the queue and size from data

The two numbers that matter are how many builds are waiting and how long they wait. The Metrics plugin and the Prometheus metrics plugin expose queue size, executor counts and build durations, which you can graph in Grafana. If the queue grows at the same hours each day, raise caps or pre-warm agents for those hours. If agents sit idle, lower the idle timeout or the minimum pool size. Our post on CI/CD monitoring tools covers the dashboards.

Signals and responses

What you seeLikely causeWhat to change
Builds wait with "waiting for next available executor"Cap reached or no agent matches the labelCheck the label, then raise the cap
Agent pods stay PendingCluster has no room, or requests are too largeEnable node autoscaling, reduce requests
Agents start but never connectController URL, tunnel port or WebSocket setting wrongRead the jnlp container logs
Builds slow after moving to ephemeral agentsDependencies downloaded every timeAdd a repository proxy or bake images
Controller CPU high with few builds runningHeavy Groovy logic or large log outputMove logic into shell steps; trim logs

When agents are not the bottleneck

A single controller has limits of its own: thousands of jobs, large build histories and many plugins all add load. If the controller is the bottleneck, split by team or product into several controllers, discard old builds, and keep plugins minimal. CloudBees publishes guidance on controller sizing for its commercial distribution; open source Jenkins users usually split controllers and manage them with Configuration as Code.

Choosing between pods and VMs

Both models scale; the choice depends on what your builds need from the machine.

NeedKubernetes pod agentsCloud VM agents
Start timeSeconds when the image is cached on the nodeUsually a minute or more for boot and connect
Building container imagesUse rootless tools such as BuildKit in rootless mode or Buildah; avoid privileged pods where possibleDocker daemon available directly
Full OS access, kernel modules, nested virtualizationLimitedAvailable
Packing many small buildsEfficient: the scheduler bin-packs podsOne build per VM wastes capacity on small jobs
Operating effortNeeds a cluster you already run wellNeeds AMIs or images and an Auto Scaling group

Many teams run both: a Kubernetes cloud for most pipelines and a small VM pool, selected by label, for jobs that need a full machine. Note that the Kaniko project was archived by Google in 2025, so check the current status of any image build tool before standardizing on it.

Configuring clouds as code

Clicking through cloud settings on the controller does not scale past one instance. The Configuration as Code (JCasC) plugin lets you define the Kubernetes cloud, pod templates, caps and timeouts in YAML that lives in Git.

jenkins:
  numExecutors: 0
  clouds:
    - kubernetes:
        name: "k8s"
        namespace: "jenkins-agents"
        jenkinsUrl: "http://jenkins.jenkins.svc.cluster.local:8080"
        containerCapStr: "50"
        webSocket: true
        templates:
          - name: "default"
            label: "linux"
            idleMinutes: 0
            containers:
              - name: "jnlp"
                image: "jenkins/inbound-agent:latest"
                resourceRequestCpu: "500m"
                resourceRequestMemory: "512Mi"

Field names follow the plugin's data model and can change between plugin versions. Export your current configuration from Manage Jenkins, Configuration as Code, View Configuration, and use that as the starting point instead of copying examples. Pin the inbound agent image to a specific version in production rather than latest.

A rollout plan for moving off static agents

  1. Inventory current agents and the tools each job relies on. Missing tools are the main reason jobs fail after migration.
  2. Build one or two agent images (or pod templates) that cover most jobs, plus labels for the exceptions.
  3. Move low-risk jobs first, such as pull request builds, and compare build times and failure rates for a week.
  4. Add the dependency cache or repository proxy before moving large builds, or build times will jump.
  5. Set caps from observed peak concurrency plus headroom, then reduce static agents as the queue stays healthy.
  6. Keep one static agent for jobs that genuinely need special hardware, and label it clearly.

Learn Jenkins hands-on

Our Jenkins training includes lab work on distributed builds, the Kubernetes plugin and pipeline design, in the Kharadi, Pune classroom or live online. If you are preparing for interviews, read 100+ Jenkins interview questions.

Frequently asked questions

What is a Jenkins agent?

An agent is a machine or container that connects to the Jenkins controller and runs build steps. The controller schedules work and stores results; agents do the actual building and testing.

How many executors should an agent have?

For ephemeral agents, one. For static agents, start with one executor per CPU core available to builds and reduce it if builds contend for disk or memory.

Should the built-in node run builds?

No. Set its executors to zero. Builds on the controller can access JENKINS_HOME and slow down the web UI and scheduler.

What is the jnlp container in a Kubernetes agent pod?

It runs the Jenkins inbound agent, which connects the pod to the controller. Your build tools run in other containers in the same pod and share the workspace volume.

Can Jenkins agents run on spot instances?

Yes. Use the EC2 Fleet plugin with a Spot Fleet or Auto Scaling group, or a spot node pool in Kubernetes. Retry interruptible stages and keep deployments on on-demand capacity.

How do I know if I need more agents?

Watch queue length and queue wait time. Consistent waits during working hours mean you need more capacity or higher caps; idle agents mean you can scale down.

Is WebSocket better than the TCP agent port?

It is simpler to run through proxies and ingress because it uses the controller's HTTP port. Both work; choose WebSocket when opening an extra port is a problem.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.