12 Kubernetes Resource Optimization Techniques
12 Kubernetes resource optimization techniques with YAML: data-based requests, CPU limits, QoS, in-place resize, HPA tuning, bin packing, spot nodes.
Kubernetes resource optimization means giving each workload the CPU and memory it actually uses, no more and no less, and then packing those workloads onto as few nodes as safely possible. The core techniques are data-based requests, a deliberate limits policy, QoS awareness, autoscaling with tuned behavior, namespace guardrails, and node consolidation that respects disruption budgets.
This guide covers the configuration side: the YAML fields, scheduler settings and runtime flags you change to cut waste. If you want the tools that measure and automate this work (Metrics Server, VPA, Goldilocks, KEDA, Karpenter, OpenCost), read our companion post on the top 12 Kubernetes tools for resource optimization. For the control plane pieces referenced below, see Kubernetes architecture: 12 components explained.
Key takeaways
- The scheduler places pods by requests, not by real usage. Inflated requests are the most common source of wasted nodes.
- Set requests from observed usage (a high percentile over at least a week), then review them on a schedule.
- Memory limits protect nodes; CPU limits cause throttling. Many teams set memory limits on everything and CPU limits only where they need hard isolation.
- In-place pod resize became stable in Kubernetes v1.35 (December 2025), so CPU and memory can change on a running pod without recreating it.
- Autoscalers only save money if their behavior is tuned. Default HPA scale-down waits 300 seconds, which is fine for most services.
- Consolidation (fewer, fuller nodes) is where the savings show up on the bill, but it needs PodDisruptionBudgets and spread rules to stay safe.
Why clusters waste resources
When you create a pod, the scheduler adds up the resources.requests of its containers and looks for a node with that much unreserved capacity. Actual usage plays no part in that decision. A pod that requests 2 CPU and uses 0.2 CPU still blocks 2 CPU on its node. Multiply that by a few hundred pods and you pay for nodes that sit mostly idle while the cluster reports itself as "full".
Limits are enforced by the kubelet and the container runtime through Linux cgroups. A container that exceeds its memory limit is OOM-killed. A container that hits its CPU limit is throttled for the rest of the scheduling period. Both behaviors matter for the techniques below.
| Setting | Who uses it | Effect when wrong |
|---|---|---|
| CPU request | Scheduler, CFS weight | Too high: idle nodes. Too low: noisy neighbours and slow pods under contention. |
| Memory request | Scheduler, eviction ranking | Too high: idle nodes. Too low: pod evicted first under node memory pressure. |
| CPU limit | cgroup quota | Too low: throttling and latency spikes even when the node has free CPU. |
| Memory limit | cgroup memory max | Too low: OOMKilled restarts. Missing: one leak can pressure the whole node. |
The 12 techniques at a glance
| # | Technique | Main saving | Effort |
|---|---|---|---|
| 1 | Set requests from measured usage | Node count | Medium |
| 2 | Choose a limits policy on purpose | Latency, stability | Low |
| 3 | Use QoS classes deliberately | Eviction safety | Low |
| 4 | Resize pods in place | Restarts avoided | Medium |
| 5 | Tune HPA behavior | Replica count | Low |
| 6 | Scale idle workloads to zero | Off-hours spend | Medium |
| 7 | Namespace defaults and quotas | Runaway requests | Low |
| 8 | Bin packing with scheduler scoring | Node count | Medium |
| 9 | Consolidate nodes safely with PDBs | Node count | Medium |
| 10 | Run interruptible work on spot capacity | Price per node | Medium |
| 11 | Make runtimes container aware | Memory headroom | Low |
| 12 | Clean up finished and orphaned objects | Storage, IPs, clutter | Low |
Sizing pods correctly
1. Set requests from measured usage
Start with data. Collect per-container CPU and memory usage for at least seven days so you see weekday and weekend patterns. A common starting rule is: CPU request near the 90th or 95th percentile of usage, memory request near the peak working set plus a small buffer. Memory deserves the bigger buffer because running out of it kills the container, while running short of CPU only slows it.
A quick look with Metrics Server installed:
kubectl top pods -n payments --containers
kubectl get pods -n payments -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory
kubectl top shows a point in time only. For percentiles, query Prometheus, for example the 95th percentile of rate(container_cpu_usage_seconds_total[5m]) over seven days per container. The Vertical Pod Autoscaler in recommendation-only mode (updateMode: "Off") produces the same kind of numbers without changing anything.
resources:
requests:
cpu: "250m"
memory: "384Mi"
limits:
memory: "512Mi"
Review requests every release cycle or at least monthly. Code changes shift usage, and a request that was right in March may be double what the service needs in September.
2. Choose a limits policy on purpose
CPU limits are enforced by the Linux CFS quota. A container with a 500m limit gets 50 ms of CPU time per 100 ms period. If a multi-threaded process burns that in the first 20 ms, it waits for the remaining 80 ms even when the node has idle cores. That shows up as tail latency that does not match CPU graphs.
A policy many teams adopt:
- Always set a memory limit, usually equal to or a little above the memory request.
- Set CPU requests on everything so the scheduler and CFS weights work.
- Set CPU limits only for workloads that need hard isolation (multi-tenant clusters, untrusted batch jobs) or must be in the Guaranteed QoS class.
Watch container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total. A throttling ratio above a few percent on a latency-sensitive service is a sign the CPU limit is too tight.
3. Use QoS classes deliberately
Kubernetes assigns every pod one of three Quality of Service classes based on its resource settings. The kubelet uses the class, along with usage compared to requests, to decide which pods to evict when a node runs short of memory.
| QoS class | Rule | Good for |
|---|---|---|
| Guaranteed | Every container has CPU and memory requests equal to limits | Databases, critical control services |
| Burstable | At least one request or limit set, not Guaranteed | Most stateless services |
| BestEffort | No requests or limits at all | Almost nothing in production |
Check a pod's class with kubectl get pod NAME -o jsonpath='{.status.qosClass}'. Combine QoS with a PriorityClass so that when the cluster is short of capacity, the scheduler can preempt low-priority batch pods instead of leaving checkout pods pending.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: batch-low
value: 1000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Interruptible batch work"
4. Resize pods in place
In-place pod resize graduated to stable in Kubernetes v1.35. CPU and memory requests and limits on a running pod are now mutable through the pod's resize subresource, often without a container restart. Before this, changing resources meant deleting and recreating the pod, which discouraged frequent rightsizing.
spec:
containers:
- name: api
image: registry.example.com/api:1.8.2
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired
- resourceName: memory
restartPolicy: RestartContainer
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
memory: "512Mi"
kubectl patch pod api-7d9f8 -n payments --subresource resize \
--patch '{"spec":{"containers":[{"name":"api","resources":{"requests":{"cpu":"300m"}}}]}}'
Limits documented by the Kubernetes project: resize does not work with swap, the static CPU Manager or the static Memory Manager, and only CPU and memory can change. Memory decreases are allowed only when current usage is below the new limit, and that check is best effort. The VPA InPlaceOrRecreate update mode, which uses this feature, was beta at the v1.35 announcement, so test it before relying on it.
Scaling with demand
5. Tune HPA behavior
The Horizontal Pod Autoscaler computes replicas from the ratio of current to target metric. Because the target is a percentage of the request, wrong requests make HPA wrong too, so do technique 1 first. Then tune the behavior block in autoscaling/v2:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 2
periodSeconds: 60
Scale up fast, scale down gently. The 300-second scale-down window is the default; shortening it saves a little money but risks flapping. Do not run HPA and VPA on the same metric (CPU or memory) for the same workload, because they will fight. Our post on 12 scaling strategies for Kubernetes applications covers custom and external metrics in more depth.
6. Scale idle workloads to zero
HPA cannot go below one replica by default. Dev, preview and QA environments often sit idle for nights and weekends. Options:
- A CronJob or CI schedule that runs
kubectl scale deployment --replicas=0in the evening and restores replicas in the morning. - KEDA, which can scale a Deployment to zero when its event source (queue length, HTTP requests through its add-on, cron trigger) is idle.
- Deleting short-lived preview namespaces automatically when the pull request closes.
Once pods are gone, the node autoscaler can remove the nodes. Scaling pods to zero without node autoscaling saves nothing.
Guardrails for teams
7. Namespace defaults and quotas
Without defaults, a developer who forgets the resources block creates a BestEffort pod. Without quotas, one team can request half the cluster. A LimitRange fills in defaults and caps per-container values; a ResourceQuota caps the namespace total.
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: team-a
spec:
limits:
- type: Container
defaultRequest:
cpu: "100m"
memory: "128Mi"
default:
memory: "256Mi"
max:
cpu: "2"
memory: "2Gi"
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-a-quota
namespace: team-a
spec:
hard:
requests.cpu: "20"
requests.memory: "40Gi"
limits.memory: "60Gi"
pods: "100"
Note that a LimitRange default sets a limit. If you only want default memory limits, leave CPU out of default as shown. Pair quotas with a policy engine rule (Kyverno or Gatekeeper) that rejects pods without requests, so the defaults are a safety net rather than the norm.
Packing nodes efficiently
8. Bin packing with scheduler scoring
The default scheduler scoring spreads pods across nodes (LeastAllocated), which is good for resilience and bad for consolidation. The NodeResourcesFit plugin can instead prefer fuller nodes with the MostAllocated strategy, so empty nodes stay empty and the autoscaler can remove them.
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: bin-packing-scheduler
pluginConfig:
- name: NodeResourcesFit
args:
scoringStrategy:
type: MostAllocated
resources:
- name: cpu
weight: 1
- name: memory
weight: 1
On managed services you usually cannot edit the default scheduler, so run this as a second scheduler profile and opt workloads in with schedulerName, or rely on the node autoscaler's own consolidation. Keep topologySpreadConstraints on critical Deployments so bin packing never puts all replicas on one node or zone.
9. Consolidate nodes safely with PDBs
Savings appear on the bill only when nodes are removed. Cluster Autoscaler removes nodes whose pods fit elsewhere; Karpenter v1 does the same with consolidationPolicy: WhenEmptyOrUnderutilized on a NodePool. Both drain nodes by evicting pods, and both respect PodDisruptionBudgets.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: api
Common blockers that keep half-empty nodes alive: PDBs with maxUnavailable: 0, pods with local storage, bare pods without a controller, and pods annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false". Audit these quarterly. The cluster stability add-ons post covers the Descheduler, which rebalances pods after the fact.
10. Run interruptible work on spot capacity
AWS Spot, Azure Spot VMs and Google Cloud Spot VMs sell spare capacity at a discount, with the right to reclaim it at short notice (AWS gives a two-minute interruption notice). Put stateless, replicated or retryable workloads there and keep stateful systems on on-demand nodes.
# Spot node group is tainted: spot=true:NoSchedule
tolerations:
- key: "spot"
operator: "Equal"
value: "true"
effect: "NoSchedule"
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: "karpenter.sh/capacity-type"
operator: In
values: ["spot"]
Use several instance types and zones so one capacity pool drying up does not stop scaling, set terminationGracePeriodSeconds to something your app can finish in, and run a node termination handler (or Karpenter's interruption handling) so pods drain before the instance disappears.
Runtime and housekeeping
11. Make runtimes container aware
Some runtimes size themselves from the host, not the cgroup. That leads either to OOM kills or to engineers padding memory requests "just in case".
- Java: modern JDKs read cgroup limits. Use
-XX:MaxRAMPercentage=75.0instead of a fixed-Xmxso heap follows the limit and leaves room for metaspace and threads. - Go: set
GOMEMLIMITa little below the memory limit so the garbage collector works harder before the kernel kills the process. Go 1.25 and later also respect cgroup CPU limits forGOMAXPROCSby default. - Node.js: set
--max-old-space-sizeto fit within the limit.
env:
- name: JAVA_TOOL_OPTIONS
value: "-XX:MaxRAMPercentage=75.0"
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
Smaller images also help indirectly: faster pulls mean new nodes become useful sooner, so autoscalers need less headroom. Set ephemeral-storage requests and limits for pods that write scratch files, otherwise disk pressure evictions can hit unrelated pods.
12. Clean up finished and orphaned objects
Completed Jobs, failed pods, unattached PersistentVolumes and forgotten LoadBalancer Services all cost something: storage, public IPs, etcd space or simply confusion during audits.
apiVersion: batch/v1
kind: Job
metadata:
name: report
spec:
ttlSecondsAfterFinished: 3600
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: report
image: registry.example.com/report:2.1
resources:
requests:
cpu: "500m"
memory: "256Mi"
limits:
memory: "512Mi"
kubectl get pv | grep Released
kubectl get svc -A --field-selector spec.type=LoadBalancer
kubectl get pods -A --field-selector status.phase=Failed
Set CronJob successfulJobsHistoryLimit and failedJobsHistoryLimit, and check whether a reclaimPolicy: Retain StorageClass is leaving disks behind after namespaces are deleted.
A rollout order that works
- Week 1: install metrics, record baseline node count and cost per namespace, list the top 20 workloads by requested CPU.
- Week 2: rightsize those 20 from percentiles, add LimitRange and ResourceQuota to every team namespace.
- Week 3: fix HPA targets and behavior, add PDBs, remove consolidation blockers.
- Week 4: enable node consolidation, move batch and stateless work to spot, schedule dev environments to zero.
- Monthly: compare requested versus used CPU and memory per namespace and repeat.
Track two numbers: requested-to-used ratio per namespace and allocatable utilisation per node pool. If both move in the right direction while error rates and latency hold, the work is paying off. When something breaks during rightsizing, the 15 common pod failures and fixes guide helps you tell OOMKilled from eviction from throttling.
Common mistakes
- Copying one resource block into every Helm chart regardless of workload.
- Setting requests from a single
kubectl topreading taken at a quiet moment. - Tight CPU limits on latency-sensitive services, then blaming the network for slow responses.
- Running HPA and VPA in auto mode on the same CPU metric.
- Enabling consolidation with no PDBs, then seeing every replica evicted at once.
- Putting databases or single-replica services on spot nodes.
Learn Kubernetes operations hands-on
Resource management, scheduling and autoscaling are core topics in the CKA exam curriculum. Our CKA training in Pune covers them with lab clusters, in the Kharadi classroom or live online.
Frequently asked questions
What is the difference between requests and limits in Kubernetes?
A request is the amount of CPU or memory the scheduler reserves for a container when placing it on a node. A limit is the maximum the container may use at runtime. Exceeding a memory limit gets the container OOM-killed; reaching a CPU limit causes throttling.
Should I set CPU limits on every pod?
Not necessarily. CPU limits can throttle a container even when the node has spare CPU. Many teams set CPU requests everywhere, memory limits everywhere, and CPU limits only where they need hard isolation or Guaranteed QoS.
How do I find the right request values?
Measure real usage for at least a week and use a high percentile (around p90 to p95) for CPU and the peak working set plus a buffer for memory. VPA in recommendation-only mode or Prometheus queries give you those numbers.
Can I change pod resources without restarting?
Yes, from Kubernetes v1.35 in-place pod resize is stable. You patch the pod's resize subresource and, depending on the container's resizePolicy, CPU and memory change without recreating the pod. It does not work with swap or the static CPU and Memory Managers.
What are Kubernetes QoS classes?
Guaranteed (requests equal limits for CPU and memory in every container), Burstable (some requests or limits set) and BestEffort (none set). The kubelet evicts BestEffort pods first and Guaranteed pods last when a node runs out of memory.
Why does my cluster have idle nodes the autoscaler will not remove?
Usually because a pod on the node cannot be moved: a PodDisruptionBudget allows zero disruptions, the pod uses local storage, it has no controller, or it carries a safe-to-evict false annotation. Fix those and consolidation can proceed.
Is spot capacity safe for Kubernetes production workloads?
For stateless, replicated or retryable workloads, yes, with several instance types, PDBs and interruption handling. Keep databases, single replicas and long uninterruptible jobs on on-demand nodes.
How is this post different from the Kubernetes optimization tools post?
This post covers the configuration techniques (YAML fields, scheduler scoring, runtime flags). The tools post covers the software that measures and automates them, such as VPA, Goldilocks, KEDA, Karpenter and OpenCost.
References
- Kubernetes: Resource management for pods and containers
- Kubernetes: Pod Quality of Service classes
- Kubernetes blog: In-place pod resize graduates to stable (v1.35)
- Kubernetes: Horizontal Pod Autoscaling
- Kubernetes: Limit Ranges
- Kubernetes: Resource bin packing
- Karpenter: Disruption and consolidation
- AWS: Spot Instance interruption notices
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0