12 Kubernetes Resource Optimization Techniques

12 Kubernetes resource optimization techniques with YAML: data-based requests, CPU limits, QoS, in-place resize, HPA tuning, bin packing, spot nodes.

Dec 30, 2025 - 18:02
Nov 8, 2026 - 22:38
 0  109
12 Kubernetes Resource Optimization Techniques

Kubernetes resource optimization means giving each workload the CPU and memory it actually uses, no more and no less, and then packing those workloads onto as few nodes as safely possible. The core techniques are data-based requests, a deliberate limits policy, QoS awareness, autoscaling with tuned behavior, namespace guardrails, and node consolidation that respects disruption budgets.

This guide covers the configuration side: the YAML fields, scheduler settings and runtime flags you change to cut waste. If you want the tools that measure and automate this work (Metrics Server, VPA, Goldilocks, KEDA, Karpenter, OpenCost), read our companion post on the top 12 Kubernetes tools for resource optimization. For the control plane pieces referenced below, see Kubernetes architecture: 12 components explained.

Key takeaways

  • The scheduler places pods by requests, not by real usage. Inflated requests are the most common source of wasted nodes.
  • Set requests from observed usage (a high percentile over at least a week), then review them on a schedule.
  • Memory limits protect nodes; CPU limits cause throttling. Many teams set memory limits on everything and CPU limits only where they need hard isolation.
  • In-place pod resize became stable in Kubernetes v1.35 (December 2025), so CPU and memory can change on a running pod without recreating it.
  • Autoscalers only save money if their behavior is tuned. Default HPA scale-down waits 300 seconds, which is fine for most services.
  • Consolidation (fewer, fuller nodes) is where the savings show up on the bill, but it needs PodDisruptionBudgets and spread rules to stay safe.

Why clusters waste resources

When you create a pod, the scheduler adds up the resources.requests of its containers and looks for a node with that much unreserved capacity. Actual usage plays no part in that decision. A pod that requests 2 CPU and uses 0.2 CPU still blocks 2 CPU on its node. Multiply that by a few hundred pods and you pay for nodes that sit mostly idle while the cluster reports itself as "full".

Limits are enforced by the kubelet and the container runtime through Linux cgroups. A container that exceeds its memory limit is OOM-killed. A container that hits its CPU limit is throttled for the rest of the scheduling period. Both behaviors matter for the techniques below.

SettingWho uses itEffect when wrong
CPU requestScheduler, CFS weightToo high: idle nodes. Too low: noisy neighbours and slow pods under contention.
Memory requestScheduler, eviction rankingToo high: idle nodes. Too low: pod evicted first under node memory pressure.
CPU limitcgroup quotaToo low: throttling and latency spikes even when the node has free CPU.
Memory limitcgroup memory maxToo low: OOMKilled restarts. Missing: one leak can pressure the whole node.

The 12 techniques at a glance

#TechniqueMain savingEffort
1Set requests from measured usageNode countMedium
2Choose a limits policy on purposeLatency, stabilityLow
3Use QoS classes deliberatelyEviction safetyLow
4Resize pods in placeRestarts avoidedMedium
5Tune HPA behaviorReplica countLow
6Scale idle workloads to zeroOff-hours spendMedium
7Namespace defaults and quotasRunaway requestsLow
8Bin packing with scheduler scoringNode countMedium
9Consolidate nodes safely with PDBsNode countMedium
10Run interruptible work on spot capacityPrice per nodeMedium
11Make runtimes container awareMemory headroomLow
12Clean up finished and orphaned objectsStorage, IPs, clutterLow

Sizing pods correctly

1. Set requests from measured usage

Start with data. Collect per-container CPU and memory usage for at least seven days so you see weekday and weekend patterns. A common starting rule is: CPU request near the 90th or 95th percentile of usage, memory request near the peak working set plus a small buffer. Memory deserves the bigger buffer because running out of it kills the container, while running short of CPU only slows it.

A quick look with Metrics Server installed:

kubectl top pods -n payments --containers
kubectl get pods -n payments -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory

kubectl top shows a point in time only. For percentiles, query Prometheus, for example the 95th percentile of rate(container_cpu_usage_seconds_total[5m]) over seven days per container. The Vertical Pod Autoscaler in recommendation-only mode (updateMode: "Off") produces the same kind of numbers without changing anything.

resources:
  requests:
    cpu: "250m"
    memory: "384Mi"
  limits:
    memory: "512Mi"

Review requests every release cycle or at least monthly. Code changes shift usage, and a request that was right in March may be double what the service needs in September.

2. Choose a limits policy on purpose

CPU limits are enforced by the Linux CFS quota. A container with a 500m limit gets 50 ms of CPU time per 100 ms period. If a multi-threaded process burns that in the first 20 ms, it waits for the remaining 80 ms even when the node has idle cores. That shows up as tail latency that does not match CPU graphs.

A policy many teams adopt:

  • Always set a memory limit, usually equal to or a little above the memory request.
  • Set CPU requests on everything so the scheduler and CFS weights work.
  • Set CPU limits only for workloads that need hard isolation (multi-tenant clusters, untrusted batch jobs) or must be in the Guaranteed QoS class.

Watch container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total. A throttling ratio above a few percent on a latency-sensitive service is a sign the CPU limit is too tight.

3. Use QoS classes deliberately

Kubernetes assigns every pod one of three Quality of Service classes based on its resource settings. The kubelet uses the class, along with usage compared to requests, to decide which pods to evict when a node runs short of memory.

QoS classRuleGood for
GuaranteedEvery container has CPU and memory requests equal to limitsDatabases, critical control services
BurstableAt least one request or limit set, not GuaranteedMost stateless services
BestEffortNo requests or limits at allAlmost nothing in production

Check a pod's class with kubectl get pod NAME -o jsonpath='{.status.qosClass}'. Combine QoS with a PriorityClass so that when the cluster is short of capacity, the scheduler can preempt low-priority batch pods instead of leaving checkout pods pending.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-low
value: 1000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Interruptible batch work"

4. Resize pods in place

In-place pod resize graduated to stable in Kubernetes v1.35. CPU and memory requests and limits on a running pod are now mutable through the pod's resize subresource, often without a container restart. Before this, changing resources meant deleting and recreating the pod, which discouraged frequent rightsizing.

spec:
  containers:
  - name: api
    image: registry.example.com/api:1.8.2
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"
      limits:
        memory: "512Mi"
kubectl patch pod api-7d9f8 -n payments --subresource resize \
  --patch '{"spec":{"containers":[{"name":"api","resources":{"requests":{"cpu":"300m"}}}]}}'

Limits documented by the Kubernetes project: resize does not work with swap, the static CPU Manager or the static Memory Manager, and only CPU and memory can change. Memory decreases are allowed only when current usage is below the new limit, and that check is best effort. The VPA InPlaceOrRecreate update mode, which uses this feature, was beta at the v1.35 announcement, so test it before relying on it.

Scaling with demand

5. Tune HPA behavior

The Horizontal Pod Autoscaler computes replicas from the ratio of current to target metric. Because the target is a percentage of the request, wrong requests make HPA wrong too, so do technique 1 first. Then tune the behavior block in autoscaling/v2:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

Scale up fast, scale down gently. The 300-second scale-down window is the default; shortening it saves a little money but risks flapping. Do not run HPA and VPA on the same metric (CPU or memory) for the same workload, because they will fight. Our post on 12 scaling strategies for Kubernetes applications covers custom and external metrics in more depth.

6. Scale idle workloads to zero

HPA cannot go below one replica by default. Dev, preview and QA environments often sit idle for nights and weekends. Options:

  • A CronJob or CI schedule that runs kubectl scale deployment --replicas=0 in the evening and restores replicas in the morning.
  • KEDA, which can scale a Deployment to zero when its event source (queue length, HTTP requests through its add-on, cron trigger) is idle.
  • Deleting short-lived preview namespaces automatically when the pull request closes.

Once pods are gone, the node autoscaler can remove the nodes. Scaling pods to zero without node autoscaling saves nothing.

Guardrails for teams

7. Namespace defaults and quotas

Without defaults, a developer who forgets the resources block creates a BestEffort pod. Without quotas, one team can request half the cluster. A LimitRange fills in defaults and caps per-container values; a ResourceQuota caps the namespace total.

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-a
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    default:
      memory: "256Mi"
    max:
      cpu: "2"
      memory: "2Gi"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "40Gi"
    limits.memory: "60Gi"
    pods: "100"

Note that a LimitRange default sets a limit. If you only want default memory limits, leave CPU out of default as shown. Pair quotas with a policy engine rule (Kyverno or Gatekeeper) that rejects pods without requests, so the defaults are a safety net rather than the norm.

Packing nodes efficiently

8. Bin packing with scheduler scoring

The default scheduler scoring spreads pods across nodes (LeastAllocated), which is good for resilience and bad for consolidation. The NodeResourcesFit plugin can instead prefer fuller nodes with the MostAllocated strategy, so empty nodes stay empty and the autoscaler can remove them.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: bin-packing-scheduler
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

On managed services you usually cannot edit the default scheduler, so run this as a second scheduler profile and opt workloads in with schedulerName, or rely on the node autoscaler's own consolidation. Keep topologySpreadConstraints on critical Deployments so bin packing never puts all replicas on one node or zone.

9. Consolidate nodes safely with PDBs

Savings appear on the bill only when nodes are removed. Cluster Autoscaler removes nodes whose pods fit elsewhere; Karpenter v1 does the same with consolidationPolicy: WhenEmptyOrUnderutilized on a NodePool. Both drain nodes by evicting pods, and both respect PodDisruptionBudgets.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

Common blockers that keep half-empty nodes alive: PDBs with maxUnavailable: 0, pods with local storage, bare pods without a controller, and pods annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false". Audit these quarterly. The cluster stability add-ons post covers the Descheduler, which rebalances pods after the fact.

10. Run interruptible work on spot capacity

AWS Spot, Azure Spot VMs and Google Cloud Spot VMs sell spare capacity at a discount, with the right to reclaim it at short notice (AWS gives a two-minute interruption notice). Put stateless, replicated or retryable workloads there and keep stateful systems on on-demand nodes.

# Spot node group is tainted: spot=true:NoSchedule
tolerations:
- key: "spot"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 80
      preference:
        matchExpressions:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot"]

Use several instance types and zones so one capacity pool drying up does not stop scaling, set terminationGracePeriodSeconds to something your app can finish in, and run a node termination handler (or Karpenter's interruption handling) so pods drain before the instance disappears.

Runtime and housekeeping

11. Make runtimes container aware

Some runtimes size themselves from the host, not the cgroup. That leads either to OOM kills or to engineers padding memory requests "just in case".

  • Java: modern JDKs read cgroup limits. Use -XX:MaxRAMPercentage=75.0 instead of a fixed -Xmx so heap follows the limit and leaves room for metaspace and threads.
  • Go: set GOMEMLIMIT a little below the memory limit so the garbage collector works harder before the kernel kills the process. Go 1.25 and later also respect cgroup CPU limits for GOMAXPROCS by default.
  • Node.js: set --max-old-space-size to fit within the limit.
env:
- name: JAVA_TOOL_OPTIONS
  value: "-XX:MaxRAMPercentage=75.0"
- name: GOMEMLIMIT
  valueFrom:
    resourceFieldRef:
      resource: limits.memory

Smaller images also help indirectly: faster pulls mean new nodes become useful sooner, so autoscalers need less headroom. Set ephemeral-storage requests and limits for pods that write scratch files, otherwise disk pressure evictions can hit unrelated pods.

12. Clean up finished and orphaned objects

Completed Jobs, failed pods, unattached PersistentVolumes and forgotten LoadBalancer Services all cost something: storage, public IPs, etcd space or simply confusion during audits.

apiVersion: batch/v1
kind: Job
metadata:
  name: report
spec:
  ttlSecondsAfterFinished: 3600
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: report
        image: registry.example.com/report:2.1
        resources:
          requests:
            cpu: "500m"
            memory: "256Mi"
          limits:
            memory: "512Mi"
kubectl get pv | grep Released
kubectl get svc -A --field-selector spec.type=LoadBalancer
kubectl get pods -A --field-selector status.phase=Failed

Set CronJob successfulJobsHistoryLimit and failedJobsHistoryLimit, and check whether a reclaimPolicy: Retain StorageClass is leaving disks behind after namespaces are deleted.

A rollout order that works

  1. Week 1: install metrics, record baseline node count and cost per namespace, list the top 20 workloads by requested CPU.
  2. Week 2: rightsize those 20 from percentiles, add LimitRange and ResourceQuota to every team namespace.
  3. Week 3: fix HPA targets and behavior, add PDBs, remove consolidation blockers.
  4. Week 4: enable node consolidation, move batch and stateless work to spot, schedule dev environments to zero.
  5. Monthly: compare requested versus used CPU and memory per namespace and repeat.

Track two numbers: requested-to-used ratio per namespace and allocatable utilisation per node pool. If both move in the right direction while error rates and latency hold, the work is paying off. When something breaks during rightsizing, the 15 common pod failures and fixes guide helps you tell OOMKilled from eviction from throttling.

Common mistakes

  • Copying one resource block into every Helm chart regardless of workload.
  • Setting requests from a single kubectl top reading taken at a quiet moment.
  • Tight CPU limits on latency-sensitive services, then blaming the network for slow responses.
  • Running HPA and VPA in auto mode on the same CPU metric.
  • Enabling consolidation with no PDBs, then seeing every replica evicted at once.
  • Putting databases or single-replica services on spot nodes.

Learn Kubernetes operations hands-on

Resource management, scheduling and autoscaling are core topics in the CKA exam curriculum. Our CKA training in Pune covers them with lab clusters, in the Kharadi classroom or live online.

Frequently asked questions

What is the difference between requests and limits in Kubernetes?

A request is the amount of CPU or memory the scheduler reserves for a container when placing it on a node. A limit is the maximum the container may use at runtime. Exceeding a memory limit gets the container OOM-killed; reaching a CPU limit causes throttling.

Should I set CPU limits on every pod?

Not necessarily. CPU limits can throttle a container even when the node has spare CPU. Many teams set CPU requests everywhere, memory limits everywhere, and CPU limits only where they need hard isolation or Guaranteed QoS.

How do I find the right request values?

Measure real usage for at least a week and use a high percentile (around p90 to p95) for CPU and the peak working set plus a buffer for memory. VPA in recommendation-only mode or Prometheus queries give you those numbers.

Can I change pod resources without restarting?

Yes, from Kubernetes v1.35 in-place pod resize is stable. You patch the pod's resize subresource and, depending on the container's resizePolicy, CPU and memory change without recreating the pod. It does not work with swap or the static CPU and Memory Managers.

What are Kubernetes QoS classes?

Guaranteed (requests equal limits for CPU and memory in every container), Burstable (some requests or limits set) and BestEffort (none set). The kubelet evicts BestEffort pods first and Guaranteed pods last when a node runs out of memory.

Why does my cluster have idle nodes the autoscaler will not remove?

Usually because a pod on the node cannot be moved: a PodDisruptionBudget allows zero disruptions, the pod uses local storage, it has no controller, or it carries a safe-to-evict false annotation. Fix those and consolidation can proceed.

Is spot capacity safe for Kubernetes production workloads?

For stateless, replicated or retryable workloads, yes, with several instance types, PDBs and interruption handling. Keep databases, single replicas and long uninterruptible jobs on on-demand nodes.

How is this post different from the Kubernetes optimization tools post?

This post covers the configuration techniques (YAML fields, scheduler scoring, runtime flags). The tools post covers the software that measures and automates them, such as VPA, Goldilocks, KEDA, Karpenter and OpenCost.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.