10 Kubernetes Node Maintenance Tips: Cordon, Drain, PDB

10 Kubernetes node maintenance tips: cordon, drain flags, PodDisruptionBudgets, stateful workloads, OS patching, kubelet upgrades and health checks.

Dec 29, 2025 - 14:52
Nov 6, 2026 - 03:32
 0  94
10 Kubernetes Node Maintenance Tips: Cordon, Drain, PDB

Safe Kubernetes node maintenance comes down to a repeatable sequence: protect workloads with PodDisruptionBudgets and enough replicas, cordon the node, drain it through the Eviction API, patch or replace it, verify health, then uncordon. Do one node or one small batch at a time, and watch the cluster between steps.

Nodes need kernel patches, OS upgrades, kubelet and container runtime updates, and sometimes new hardware. Each of these means taking a node out of service while the applications on it keep running somewhere else. This guide gives you ten practical tips, in roughly the order you apply them, with the exact kubectl commands and the settings that decide whether a drain finishes in two minutes or hangs for an hour. For how nodes fit into the control plane and kubelet picture, see our Kubernetes architecture components guide.

Key takeaways

  • kubectl cordon stops new pods landing on a node; kubectl drain cordons and then evicts existing pods.
  • Drain uses the Eviction API, which respects PodDisruptionBudgets. An eviction that would break a budget is refused and retried.
  • A single-replica Deployment always goes down during a drain. Run at least two replicas for anything that must stay up.
  • DaemonSet pods are skipped with --ignore-daemonsets; pods using emptyDir need --delete-emptydir-data, and that data is lost.
  • Set unhealthyPodEvictionPolicy: AlwaysAllow on PDBs so crash-looping pods do not block drains.
  • Upgrade the control plane before the kubelets, and keep kubelets within the supported version skew.

The maintenance sequence at a glance

StepCommand or actionWhat to check
Pre-checkkubectl get pdb -A, replica countsEvery critical app has 2+ replicas and a PDB with room to evict
Cordonkubectl cordon node-1Node shows SchedulingDisabled
Drainkubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=10mCommand exits successfully
MaintainPatch, reboot, upgrade kubelet, or replaceNode returns Ready
Uncordonkubectl uncordon node-1Pods schedule again, no new alerts

1. Make workloads disruption-ready before you touch a node

Maintenance problems usually start in the workload design, not the drain. Before any planned work, check that:

  • Every service that must stay up runs at least two replicas.
  • Replicas are spread across nodes and zones with topologySpreadConstraints or pod anti-affinity, so one node never holds all copies.
  • Readiness probes are accurate, so traffic only reaches pods that can serve it. Our guide to Kubernetes health checks covers probe settings.
  • Applications handle SIGTERM and finish in-flight requests within terminationGracePeriodSeconds (30 seconds by default).
spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: api

2. Define PodDisruptionBudgets that allow progress

A PodDisruptionBudget (PDB) limits how many pods of an application voluntary disruptions, such as drains, may take down at once. Set either minAvailable or maxUnavailable.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api
  unhealthyPodEvictionPolicy: AlwaysAllow

Two settings cause most stuck drains. The first is a budget that allows zero disruptions, for example minAvailable: 3 on a three-replica Deployment, or any minAvailable: 1 on a single replica. The drain will wait forever. The second is unhealthy pods: with the default IfHealthy policy, a pod that is not ready can only be evicted when the budget is already satisfied. The Kubernetes docs recommend AlwaysAllow so a broken application does not block node work. Run kubectl get pdb -A and look for ALLOWED DISRUPTIONS of 0 before you start.

3. Cordon first when you need to investigate

Cordoning marks a node unschedulable. Running pods stay where they are; new pods go elsewhere.

kubectl cordon node-1
kubectl get nodes
# node-1   Ready,SchedulingDisabled   ...

Cordon on its own is useful when a node shows early trouble (disk pressure, flapping network) and you want to stop it collecting new work while you look. Drain cordons automatically, so for planned maintenance you can go straight to tip 4.

4. Drain with the right flags

kubectl drain evicts pods through the Eviction API. If an eviction would break a PDB, the API server returns HTTP 429 and kubectl keeps retrying until it succeeds or hits the timeout.

kubectl drain node-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m
FlagWhy you need it
--ignore-daemonsetsDaemonSet pods would be recreated on the same node, so drain refuses without this flag
--delete-emptydir-dataAllows eviction of pods with emptyDir volumes; their scratch data is deleted
--timeoutStops the command after a set time instead of waiting indefinitely (default 0, no limit)
--grace-periodOverrides each pod's termination grace period; default -1 uses the pod's own value
--pod-selectorDrains only pods matching a label
--forceAlso deletes bare pods with no controller; they will not come back
--disable-evictionDeletes instead of evicting and bypasses PDBs; last resort only

Avoid --force and --disable-eviction as habits. If a drain hangs, find out which pod and which PDB are blocking it: kubectl get pods -A --field-selector spec.nodeName=node-1 lists what is left.

5. Treat stateful workloads with extra care

StatefulSet pods keep their identity and usually their PersistentVolumeClaims. When one is evicted it is recreated with the same name, and its volume must attach to the new node. With zonal block storage (for example a cloud disk tied to one availability zone), the pod can only reschedule in that zone, so make sure another node there has capacity.

  • Use a PDB that allows one pod at a time for quorum systems such as etcd, ZooKeeper or Kafka.
  • Check the application's own health (replication lag, leader status) between nodes, not just pod readiness.
  • Pods using local PersistentVolumes cannot move. Plan data replication at the application level or accept downtime for those replicas.

Our StatefulSet vs Deployment post explains why these pods behave differently.

6. Keep spare capacity for evicted pods

Every pod you evict needs a home. If the remaining nodes cannot fit their resource requests, evicted pods sit in Pending, and with a strict PDB the drain stalls. Before maintenance:

  • Compare requested resources with allocatable capacity: kubectl describe nodes shows both, and kubectl top nodes shows live use if metrics-server is installed.
  • On cloud clusters, add a node first (surge), then drain the old one. Managed node groups on EKS, GKE and AKS do this during upgrades with configurable surge settings.
  • If you run Cluster Autoscaler or Karpenter, confirm it can add nodes during the window and that scale-up limits are not already reached.

7. Patch the OS and reboot without surprises

Kernel and OS security updates are the most frequent reason for node work. A simple, safe loop per node:

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=10m
ssh node-1 'sudo dnf -y upgrade && sudo systemctl reboot'
kubectl wait --for=condition=Ready node/node-1 --timeout=10m
kubectl uncordon node-1

Use the package manager your distribution ships (dnf on RHEL family, apt on Debian and Ubuntu). For fleets, automate the loop. The Kubereboot project's kured DaemonSet watches for a reboot-required signal and drains, reboots and uncordons nodes one at a time. On cloud clusters, many teams replace nodes with a fresh image instead of patching in place, which also removes configuration drift.

8. Upgrade kubelet and runtime in the right order

Kubernetes defines a version skew policy: the kubelet must not be newer than the API server, and it may be up to three minor versions older (from v1.28 onward). Upgrade the control plane first, then nodes. On kubeadm clusters, the per-node steps look like this (package names depend on your repository setup):

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
# on node-1: upgrade the kubeadm package, then
sudo kubeadm upgrade node
# upgrade kubelet and kubectl packages, then
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon node-1

Container runtime upgrades (containerd or CRI-O) follow the same drain and uncordon wrapper. Read the release notes for removed APIs before every minor upgrade; our container runtime comparison covers runtime choices.

9. Verify health before moving to the next node

Do not chain drains blindly. After each node returns:

  • kubectl get nodes shows the node Ready with the expected kubelet version.
  • kubectl get pods -A | grep -v Running shows no new failures beyond normal jobs.
  • DaemonSet pods (CNI, log shipper, node exporter) are running on the node again.
  • Dashboards and alerts show normal error rate and latency for a few minutes.

If something looks off, stop. The pod troubleshooting techniques post lists the commands for digging in.

10. Automate and record the process

Once the manual steps are reliable, put them in code: an Ansible playbook with serial: 1, a pipeline job, or your cloud provider's managed node upgrade. Keep a runbook that lists the order of node pools, the PDBs that commonly block, and who to call if a stateful service misbehaves. Schedule a maintenance window for production, announce it, and record what happened so the next run is faster.

Two extra settings help here. Graceful node shutdown (kubelet shutdownGracePeriod and shutdownGracePeriodCriticalPods) lets pods terminate cleanly when a node shuts down unexpectedly, though it does not respect PDBs. And labels or taints such as node.kubernetes.io/unschedulable, which cordon adds, let your automation see which nodes are mid-maintenance.

Cordon, drain and delete compared

CommandStops new podsMoves running podsRespects PDBTypical use
kubectl cordonYesNoNot applicableInvestigation, staging work
kubectl drainYesYes, by evictionYesPlanned maintenance
kubectl delete nodeYesPods are removed by garbage collectionNoRetiring a node after it is drained

Learn it hands-on

Node drain, cluster upgrades with kubeadm and troubleshooting are part of the CKA curriculum. Our CKA training runs these tasks on real clusters, in the Kharadi, Pune classroom or live online. For keeping nodes healthy between maintenance windows, see our list of Kubernetes add-ons that improve cluster stability.

Frequently asked questions

What is the difference between kubectl cordon and kubectl drain?

Cordon marks a node unschedulable so no new pods land on it, but existing pods keep running. Drain cordons the node and then evicts its pods so they reschedule on other nodes.

Why is my kubectl drain stuck?

Usually a PodDisruptionBudget allows zero disruptions, an unhealthy pod is protected by the default IfHealthy policy, or evicted pods cannot fit elsewhere and stay Pending. Check kubectl get pdb -A and pending pods.

Does kubectl drain cause downtime?

Not if each application has multiple replicas on different nodes, accurate readiness probes and a PDB that keeps enough pods running. A single-replica workload is unavailable while its pod restarts elsewhere.

What happens to DaemonSet pods during a drain?

With --ignore-daemonsets, drain leaves them running on the node, because the DaemonSet controller would recreate them there anyway. They stop when the node shuts down and return after reboot.

Should I patch nodes in place or replace them?

On cloud clusters, replacing nodes with a new image is often simpler and avoids drift. On bare metal, in-place patching with drain, reboot and uncordon is the norm. Both follow the same drain discipline.

How many nodes can I drain at once?

As many as your spare capacity and PDBs allow. PDBs still apply across parallel drains, so the cluster will not exceed the budget, but one node or one small batch at a time is easier to watch and roll back.

What is unhealthyPodEvictionPolicy?

A PDB field that controls whether pods that are not ready can be evicted. IfHealthy (default) protects them unless the budget is met; AlwaysAllow lets drains evict them, which the Kubernetes docs recommend.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.