10 Kubernetes Node Maintenance Tips: Cordon, Drain, PDB
10 Kubernetes node maintenance tips: cordon, drain flags, PodDisruptionBudgets, stateful workloads, OS patching, kubelet upgrades and health checks.
Safe Kubernetes node maintenance comes down to a repeatable sequence: protect workloads with PodDisruptionBudgets and enough replicas, cordon the node, drain it through the Eviction API, patch or replace it, verify health, then uncordon. Do one node or one small batch at a time, and watch the cluster between steps.
Nodes need kernel patches, OS upgrades, kubelet and container runtime updates, and sometimes new hardware. Each of these means taking a node out of service while the applications on it keep running somewhere else. This guide gives you ten practical tips, in roughly the order you apply them, with the exact kubectl commands and the settings that decide whether a drain finishes in two minutes or hangs for an hour. For how nodes fit into the control plane and kubelet picture, see our Kubernetes architecture components guide.
Key takeaways
kubectl cordonstops new pods landing on a node;kubectl draincordons and then evicts existing pods.- Drain uses the Eviction API, which respects PodDisruptionBudgets. An eviction that would break a budget is refused and retried.
- A single-replica Deployment always goes down during a drain. Run at least two replicas for anything that must stay up.
- DaemonSet pods are skipped with
--ignore-daemonsets; pods usingemptyDirneed--delete-emptydir-data, and that data is lost. - Set
unhealthyPodEvictionPolicy: AlwaysAllowon PDBs so crash-looping pods do not block drains. - Upgrade the control plane before the kubelets, and keep kubelets within the supported version skew.
The maintenance sequence at a glance
| Step | Command or action | What to check |
|---|---|---|
| Pre-check | kubectl get pdb -A, replica counts | Every critical app has 2+ replicas and a PDB with room to evict |
| Cordon | kubectl cordon node-1 | Node shows SchedulingDisabled |
| Drain | kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=10m | Command exits successfully |
| Maintain | Patch, reboot, upgrade kubelet, or replace | Node returns Ready |
| Uncordon | kubectl uncordon node-1 | Pods schedule again, no new alerts |
1. Make workloads disruption-ready before you touch a node
Maintenance problems usually start in the workload design, not the drain. Before any planned work, check that:
- Every service that must stay up runs at least two replicas.
- Replicas are spread across nodes and zones with
topologySpreadConstraintsor pod anti-affinity, so one node never holds all copies. - Readiness probes are accurate, so traffic only reaches pods that can serve it. Our guide to Kubernetes health checks covers probe settings.
- Applications handle
SIGTERMand finish in-flight requests withinterminationGracePeriodSeconds(30 seconds by default).
spec:
replicas: 3
template:
spec:
terminationGracePeriodSeconds: 45
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: api
2. Define PodDisruptionBudgets that allow progress
A PodDisruptionBudget (PDB) limits how many pods of an application voluntary disruptions, such as drains, may take down at once. Set either minAvailable or maxUnavailable.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: api
unhealthyPodEvictionPolicy: AlwaysAllow
Two settings cause most stuck drains. The first is a budget that allows zero disruptions, for example minAvailable: 3 on a three-replica Deployment, or any minAvailable: 1 on a single replica. The drain will wait forever. The second is unhealthy pods: with the default IfHealthy policy, a pod that is not ready can only be evicted when the budget is already satisfied. The Kubernetes docs recommend AlwaysAllow so a broken application does not block node work. Run kubectl get pdb -A and look for ALLOWED DISRUPTIONS of 0 before you start.
3. Cordon first when you need to investigate
Cordoning marks a node unschedulable. Running pods stay where they are; new pods go elsewhere.
kubectl cordon node-1
kubectl get nodes
# node-1 Ready,SchedulingDisabled ...
Cordon on its own is useful when a node shows early trouble (disk pressure, flapping network) and you want to stop it collecting new work while you look. Drain cordons automatically, so for planned maintenance you can go straight to tip 4.
4. Drain with the right flags
kubectl drain evicts pods through the Eviction API. If an eviction would break a PDB, the API server returns HTTP 429 and kubectl keeps retrying until it succeeds or hits the timeout.
kubectl drain node-1 \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout=10m
| Flag | Why you need it |
|---|---|
--ignore-daemonsets | DaemonSet pods would be recreated on the same node, so drain refuses without this flag |
--delete-emptydir-data | Allows eviction of pods with emptyDir volumes; their scratch data is deleted |
--timeout | Stops the command after a set time instead of waiting indefinitely (default 0, no limit) |
--grace-period | Overrides each pod's termination grace period; default -1 uses the pod's own value |
--pod-selector | Drains only pods matching a label |
--force | Also deletes bare pods with no controller; they will not come back |
--disable-eviction | Deletes instead of evicting and bypasses PDBs; last resort only |
Avoid --force and --disable-eviction as habits. If a drain hangs, find out which pod and which PDB are blocking it: kubectl get pods -A --field-selector spec.nodeName=node-1 lists what is left.
5. Treat stateful workloads with extra care
StatefulSet pods keep their identity and usually their PersistentVolumeClaims. When one is evicted it is recreated with the same name, and its volume must attach to the new node. With zonal block storage (for example a cloud disk tied to one availability zone), the pod can only reschedule in that zone, so make sure another node there has capacity.
- Use a PDB that allows one pod at a time for quorum systems such as etcd, ZooKeeper or Kafka.
- Check the application's own health (replication lag, leader status) between nodes, not just pod readiness.
- Pods using local PersistentVolumes cannot move. Plan data replication at the application level or accept downtime for those replicas.
Our StatefulSet vs Deployment post explains why these pods behave differently.
6. Keep spare capacity for evicted pods
Every pod you evict needs a home. If the remaining nodes cannot fit their resource requests, evicted pods sit in Pending, and with a strict PDB the drain stalls. Before maintenance:
- Compare requested resources with allocatable capacity:
kubectl describe nodesshows both, andkubectl top nodesshows live use if metrics-server is installed. - On cloud clusters, add a node first (surge), then drain the old one. Managed node groups on EKS, GKE and AKS do this during upgrades with configurable surge settings.
- If you run Cluster Autoscaler or Karpenter, confirm it can add nodes during the window and that scale-up limits are not already reached.
7. Patch the OS and reboot without surprises
Kernel and OS security updates are the most frequent reason for node work. A simple, safe loop per node:
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=10m
ssh node-1 'sudo dnf -y upgrade && sudo systemctl reboot'
kubectl wait --for=condition=Ready node/node-1 --timeout=10m
kubectl uncordon node-1
Use the package manager your distribution ships (dnf on RHEL family, apt on Debian and Ubuntu). For fleets, automate the loop. The Kubereboot project's kured DaemonSet watches for a reboot-required signal and drains, reboots and uncordons nodes one at a time. On cloud clusters, many teams replace nodes with a fresh image instead of patching in place, which also removes configuration drift.
8. Upgrade kubelet and runtime in the right order
Kubernetes defines a version skew policy: the kubelet must not be newer than the API server, and it may be up to three minor versions older (from v1.28 onward). Upgrade the control plane first, then nodes. On kubeadm clusters, the per-node steps look like this (package names depend on your repository setup):
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
# on node-1: upgrade the kubeadm package, then
sudo kubeadm upgrade node
# upgrade kubelet and kubectl packages, then
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon node-1
Container runtime upgrades (containerd or CRI-O) follow the same drain and uncordon wrapper. Read the release notes for removed APIs before every minor upgrade; our container runtime comparison covers runtime choices.
9. Verify health before moving to the next node
Do not chain drains blindly. After each node returns:
kubectl get nodesshows the nodeReadywith the expected kubelet version.kubectl get pods -A | grep -v Runningshows no new failures beyond normal jobs.- DaemonSet pods (CNI, log shipper, node exporter) are running on the node again.
- Dashboards and alerts show normal error rate and latency for a few minutes.
If something looks off, stop. The pod troubleshooting techniques post lists the commands for digging in.
10. Automate and record the process
Once the manual steps are reliable, put them in code: an Ansible playbook with serial: 1, a pipeline job, or your cloud provider's managed node upgrade. Keep a runbook that lists the order of node pools, the PDBs that commonly block, and who to call if a stateful service misbehaves. Schedule a maintenance window for production, announce it, and record what happened so the next run is faster.
Two extra settings help here. Graceful node shutdown (kubelet shutdownGracePeriod and shutdownGracePeriodCriticalPods) lets pods terminate cleanly when a node shuts down unexpectedly, though it does not respect PDBs. And labels or taints such as node.kubernetes.io/unschedulable, which cordon adds, let your automation see which nodes are mid-maintenance.
Cordon, drain and delete compared
| Command | Stops new pods | Moves running pods | Respects PDB | Typical use |
|---|---|---|---|---|
kubectl cordon | Yes | No | Not applicable | Investigation, staging work |
kubectl drain | Yes | Yes, by eviction | Yes | Planned maintenance |
kubectl delete node | Yes | Pods are removed by garbage collection | No | Retiring a node after it is drained |
Learn it hands-on
Node drain, cluster upgrades with kubeadm and troubleshooting are part of the CKA curriculum. Our CKA training runs these tasks on real clusters, in the Kharadi, Pune classroom or live online. For keeping nodes healthy between maintenance windows, see our list of Kubernetes add-ons that improve cluster stability.
Frequently asked questions
What is the difference between kubectl cordon and kubectl drain?
Cordon marks a node unschedulable so no new pods land on it, but existing pods keep running. Drain cordons the node and then evicts its pods so they reschedule on other nodes.
Why is my kubectl drain stuck?
Usually a PodDisruptionBudget allows zero disruptions, an unhealthy pod is protected by the default IfHealthy policy, or evicted pods cannot fit elsewhere and stay Pending. Check kubectl get pdb -A and pending pods.
Does kubectl drain cause downtime?
Not if each application has multiple replicas on different nodes, accurate readiness probes and a PDB that keeps enough pods running. A single-replica workload is unavailable while its pod restarts elsewhere.
What happens to DaemonSet pods during a drain?
With --ignore-daemonsets, drain leaves them running on the node, because the DaemonSet controller would recreate them there anyway. They stop when the node shuts down and return after reboot.
Should I patch nodes in place or replace them?
On cloud clusters, replacing nodes with a new image is often simpler and avoids drift. On bare metal, in-place patching with drain, reboot and uncordon is the norm. Both follow the same drain discipline.
How many nodes can I drain at once?
As many as your spare capacity and PDBs allow. PDBs still apply across parallel drains, so the cluster will not exceed the budget, but one node or one small batch at a time is easier to watch and roll back.
What is unhealthyPodEvictionPolicy?
A PDB field that controls whether pods that are not ready can be evicted. IfHealthy (default) protects them unless the budget is met; AlwaysAllow lets drains evict them, which the Kubernetes docs recommend.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0