10 Kubernetes Health Metrics You Should Track
10 Kubernetes health metrics with exact Prometheus names, sources, PromQL and alert rules: API server, etcd, pending pods, nodes, restarts, replicas, PVCs.
The ten Kubernetes health metrics worth tracking first are API server errors and latency, etcd leader and disk latency, pending pods, node conditions, node resource saturation, container restarts, pod phase, desired versus available replicas, requests compared with allocatable capacity, and persistent volume usage. Together they show whether the control plane, nodes and workloads are healthy.
This guide gives each metric its exact Prometheus name, the component that exposes it, a PromQL query you can paste into Grafana and a sensible way to alert on it. It complements 18 observability metrics for cloud-native systems, which covers application-level signals, and 10 Kubernetes health checks every DevOps must use, which covers probes. Metric names and stability levels were checked against the Kubernetes 1.37 metrics reference, the kube-state-metrics docs and the etcd v3.6 docs in November 2026.
Key takeaways
- Health metrics come from four sources: control plane components, the kubelet (including cAdvisor), kube-state-metrics, and node_exporter. You need all four for a full picture.
- kube-state-metrics reports what Kubernetes objects say (desired replicas, pod phase, restarts); it does not measure CPU or memory.
- Check a metric's stability level before building alerts on it.
apiserver_request_totalis Stable, whileetcd_request_duration_secondsis Alpha and may change. apiserver_storage_objectsis deprecated since 1.34; useapiserver_resource_objectsinstead.- On managed clusters (EKS, AKS, GKE) you may not be able to scrape etcd or the scheduler directly. Use the provider's control plane metrics instead.
- Alert on symptoms that need action, and keep the rest on dashboards.
Where the metrics come from
| Source | What it exposes | Typical endpoint |
|---|---|---|
| kube-apiserver, scheduler, controller manager | Request rates, latencies, queue depths, scheduling attempts | /metrics on each component |
| etcd | Leader status, disk sync latency, database size | /metrics on the etcd client or metrics port |
| kubelet | Pod lifecycle, volume stats, PLEG timing | /metrics on the kubelet |
| cAdvisor (inside the kubelet) | Per-container CPU, memory, throttling | /metrics/cadvisor on the kubelet |
| kube-state-metrics | Object state: replicas, phases, restarts, conditions | Its own service |
| node_exporter | Host CPU, memory, disk and network | Port 9100 by default |
The kube-prometheus-stack Helm chart installs Prometheus, kube-state-metrics, node_exporter and default scrape configs in one go, which is how many teams start. If you are new to the pairing, read Prometheus and Grafana together first.
The ten metrics at a glance
| # | Health signal | Main metric | Source | Stability |
|---|---|---|---|---|
| 1 | API server errors and latency | apiserver_request_total, apiserver_request_duration_seconds | kube-apiserver | Stable |
| 2 | etcd leader and disk latency | etcd_server_has_leader, etcd_disk_wal_fsync_duration_seconds | etcd | etcd project metrics |
| 3 | Pods waiting to be scheduled | scheduler_pending_pods | kube-scheduler | Stable |
| 4 | Node conditions | kube_node_status_condition | kube-state-metrics | Stable |
| 5 | Node saturation | node_memory_MemAvailable_bytes, node_cpu_seconds_total | node_exporter | n/a |
| 6 | Container restarts | kube_pod_container_status_restarts_total | kube-state-metrics | Stable |
| 7 | Pod phase | kube_pod_status_phase | kube-state-metrics | Stable |
| 8 | Replicas available vs desired | kube_deployment_status_replicas_available, kube_deployment_spec_replicas | kube-state-metrics | Stable |
| 9 | Requests vs allocatable, throttling | kube_pod_container_resource_requests, kube_node_status_allocatable | kube-state-metrics, cAdvisor | Stable (KSM) |
| 10 | Persistent volume usage | kubelet_volume_stats_used_bytes | kubelet | Alpha |
1. API server error rate and latency
Every kubectl command, controller and operator goes through the API server, so its health affects the whole cluster. Both metrics are listed as Stable in the Kubernetes metrics reference.
# share of requests returning 5xx over 5 minutes
sum(rate(apiserver_request_total{code=~"5.."}[5m]))
/ sum(rate(apiserver_request_total[5m]))
# p99 latency for non-streaming verbs
histogram_quantile(0.99,
sum by (le, verb) (rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m])))
Exclude WATCH and CONNECT from latency queries: they are long-lived by design and will make the percentiles meaningless. A sustained rise in 5xx responses or a jump in mutating request latency (POST, PUT, PATCH, DELETE) is worth an alert. A rise in 429 responses means API Priority and Fairness is rejecting requests, often because a misbehaving client is flooding the server.
2. etcd leader status and disk latency
etcd stores all cluster state. It needs a leader to accept writes, and it is sensitive to slow disks because every write is synced to its write-ahead log. The etcd metrics documentation describes these names:
etcd_server_has_leader: 1 when the member sees a leader, 0 when it does not.etcd_server_leader_changes_seen_total: frequent leader changes point to network or disk trouble.etcd_disk_wal_fsync_duration_secondsandetcd_disk_backend_commit_duration_seconds: histograms of disk sync latency.
max(etcd_server_has_leader) == 0
increase(etcd_server_leader_changes_seen_total[1h]) > 3
histogram_quantile(0.99, sum by (le, instance) (rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])))
Treat "no leader" as a page-worthy alert. For fsync latency, the etcd project publishes default alert rules in the contrib/mixin directory of its repository; start from those thresholds rather than inventing your own. On managed clusters you usually cannot see etcd at all; providers expose their own control plane health signals instead. The API server's view of etcd, etcd_request_duration_seconds, is available but marked Alpha.
3. Pending pods
scheduler_pending_pods (Stable) counts pods waiting in the scheduler's queues, labelled by queue type such as active, backoff and unschedulable. A steady number in the unschedulable queue means pods cannot fit anywhere: not enough CPU or memory, taints without tolerations, affinity rules nobody can satisfy, or PersistentVolumeClaims bound to a zone with no capacity.
sum by (queue) (scheduler_pending_pods)
If you cannot scrape the scheduler, use kube-state-metrics instead: sum(kube_pod_status_phase{phase="Pending"}). Alert when pods stay pending for more than a few minutes, then run kubectl describe pod to read the scheduler's event message. If you run Cluster Autoscaler or Karpenter, pending pods are also the trigger for new nodes, so a long pending time can mean the autoscaler is failing.
4. Node conditions
The kubelet reports node conditions: Ready, MemoryPressure, DiskPressure, PIDPressure and NetworkUnavailable. kube-state-metrics turns them into kube_node_status_condition (Stable), with labels for condition and status.
# nodes not Ready
kube_node_status_condition{condition="Ready",status="true"} == 0
# nodes under any pressure
kube_node_status_condition{condition=~"MemoryPressure|DiskPressure|PIDPressure",status="true"} == 1
# cordoned nodes
kube_node_spec_unschedulable == 1
A node under memory or disk pressure starts evicting pods. A node that is NotReady for longer than the eviction timeout loses its pods to other nodes. Alert on NotReady lasting more than a few minutes, and on any pressure condition. Track cordoned nodes too, so a node left cordoned after maintenance does not quietly shrink your capacity; Kubernetes node maintenance tips covers the cordon and drain workflow.
5. Node resource saturation
Node conditions fire late, after pressure has started. node_exporter gives earlier warning:
# memory available as a fraction
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
# CPU busy fraction per node
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
# root filesystem free fraction
node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
Disk deserves special attention on nodes because container images, logs and emptyDir volumes all share it. When free space crosses the kubelet's eviction thresholds, the kubelet garbage-collects images and then evicts pods. Alert well before that point. For CPU, high utilisation alone is not a problem if workloads have correct requests; correlate it with throttling (metric 9) and latency before acting.
6. Container restarts
kube_pod_container_status_restarts_total (Stable) is a counter, so look at its increase over a window rather than its raw value:
increase(kube_pod_container_status_restarts_total[30m]) > 3
# why containers are waiting
sum by (namespace, reason) (kube_pod_container_status_waiting_reason)
kube_pod_container_status_waiting_reason (Stable) shows reasons such as CrashLoopBackOff, ImagePullBackOff and CreateContainerConfigError. To see why a container last died, kube-state-metrics offers kube_pod_container_status_last_terminated_reason, but it is marked Experimental, so do not build critical alerts on it. A reason of OOMKilled means the container hit its memory limit. Detailed fixes for each failure mode are in 15 common Kubernetes pod failures and fixes.
7. Pod phase
kube_pod_status_phase (Stable) reports each pod's phase: Pending, Running, Succeeded, Failed or Unknown.
sum by (namespace, phase) (kube_pod_status_phase)
sum(kube_pod_status_phase{phase=~"Failed|Unknown"}) by (namespace)
Failed pods from Jobs may be expected; failed pods from long-running services usually are not. Unknown usually means the control plane lost contact with the node. Phase only tells part of the story, because a Running pod can still fail its readiness probe. Pair this with kube_pod_status_ready (Stable) to count pods that are running but not ready to serve traffic.
8. Desired versus available replicas
This is the most direct "is my service healthy" signal from Kubernetes' point of view. kube-state-metrics exposes both values as Stable metrics:
kube_deployment_spec_replicas
- kube_deployment_status_replicas_available > 0
# rollout stuck: controller has not observed the latest spec
kube_deployment_status_observed_generation
!= kube_deployment_metadata_generation
Alert when the gap persists for longer than your normal rollout time (for example, 10 to 15 minutes). The same pattern exists for StatefulSets (kube_statefulset_status_replicas_ready) and DaemonSets (kube_daemonset_status_number_unavailable). For HorizontalPodAutoscalers, also watch whether current replicas sit at the configured maximum for long periods, which means the HPA wants to scale but is capped.
9. Requests versus allocatable, and CPU throttling
The scheduler places pods by their resource requests, not their actual usage. If the sum of requests on the cluster nears allocatable capacity, new pods go Pending even when nodes look idle.
# cluster-wide CPU requests as a fraction of allocatable
sum(kube_pod_container_resource_requests{resource="cpu"})
/ sum(kube_node_status_allocatable{resource="cpu"})
# share of CPU periods throttled per container (cAdvisor)
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/ sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m]))
# memory working set against limit
container_memory_working_set_bytes{container!=""}
High throttling with low average CPU means the CPU limit is too tight for bursty work. Memory working set close to the limit predicts OOM kills. Use these numbers to right-size requests and limits; 12 Kubernetes resource optimization techniques walks through the process.
10. Persistent volume usage
A full volume takes down a database just as surely as a crashed node. The kubelet exposes volume statistics per PersistentVolumeClaim:
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
# hours until full, based on the last 6 hours of growth
predict_linear(kubelet_volume_stats_available_bytes[6h], 24 * 3600) < 0
These metrics are marked Alpha in the 1.37 reference, and not every storage driver reports them, so check that you get data for your volumes before relying on the alert. The predict_linear query catches volumes that are filling fast while still below the static threshold. If the volume can be expanded, check whether the StorageClass sets allowVolumeExpansion: true.
Turning metrics into alerts
Not every metric deserves a page. A practical split:
| Severity | Examples | Response |
|---|---|---|
| Page now | etcd has no leader, API server 5xx sustained, a service with zero available replicas, PVC nearly full on a database | On-call responds immediately |
| Ticket today | Node NotReady, pods pending for 15 minutes, repeated restarts, deployment rollout stuck | Investigate during working hours |
| Dashboard only | CPU utilisation, requests vs allocatable trend, throttling ratios | Capacity planning and tuning |
Add a for: duration to every Prometheus alert rule so short blips do not wake anyone, and link each alert to a runbook. The community kubernetes-mixin project provides tested alert rules and dashboards that you can adapt. A full alert list is in 10 monitoring alerts every DevOps team should configure, and logs that complement these metrics are in 10 hidden Kubernetes logs you should check.
A starter alert rule file
groups:
- name: kubernetes-health
rules:
- alert: KubeNodeNotReady
expr: kube_node_status_condition{condition="Ready",status="true"} == 0
for: 5m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.node }} is not Ready"
- alert: KubeDeploymentReplicasMismatch
expr: kube_deployment_spec_replicas - kube_deployment_status_replicas_available > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.deployment }} has unavailable replicas"
- alert: KubePodCrashLooping
expr: increase(kube_pod_container_status_restarts_total[30m]) > 3
for: 5m
labels:
severity: warning
- alert: EtcdNoLeader
expr: etcd_server_has_leader == 0
for: 1m
labels:
severity: critical
Prometheus, PromQL, Grafana dashboards and alerting are covered in our Prometheus and Grafana training, in the Kharadi, Pune classroom and live online. Cluster operations and troubleshooting are part of the CKA course.
FAQ
What is the most important Kubernetes health metric?
For workloads, desired versus available replicas, because it directly shows whether a service has the pods it needs. For the cluster itself, API server error rate and etcd leader status, because every other component depends on them.
What is the difference between kube-state-metrics and metrics-server?
metrics-server collects current CPU and memory usage for autoscaling and kubectl top. kube-state-metrics exposes the state of Kubernetes objects, such as replicas, pod phase and restarts, as Prometheus metrics. Most monitoring setups use both.
How do I monitor etcd on EKS, AKS or GKE?
Usually you cannot scrape etcd directly, because the provider runs the control plane. Use the provider's control plane metrics and logs, and watch API server latency and errors, which reflect etcd problems.
Why are my CPU-throttled pods slow when the node is idle?
CPU limits are enforced per container through CFS quotas. A container can hit its limit during short bursts even when the node has spare CPU. Compare throttled periods with total periods and raise or remove the limit if latency suffers.
Which metrics should trigger a page?
Page on etcd losing its leader, sustained API server 5xx errors, a service with zero available replicas and storage about to fill. Node NotReady, pending pods and restarts usually go to a ticket unless they affect users.
Are these metric names stable across Kubernetes versions?
Stable metrics keep their names and labels. Alpha metrics, such as kubelet volume stats and etcd_request_duration_seconds, can change. apiserver_storage_objects is deprecated since 1.34 in favour of apiserver_resource_objects.
How do I detect a stuck deployment rollout?
Compare kube_deployment_status_observed_generation with kube_deployment_metadata_generation, and alert when desired replicas exceed available replicas for longer than a normal rollout. kubectl rollout status shows the same from the CLI.
Do I need node_exporter if I already have kube-state-metrics?
Yes. kube-state-metrics reports object state and does not measure host resources. node_exporter provides host memory, CPU, disk and network data that warn you before nodes hit pressure conditions.
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0