How Liveness and Readiness Probes Improve Reliability

What happens when liveness and readiness probes pass or fail: endpoints, restarts, rollouts, shutdown and settings that avoid cascading failures.

Aug 13, 2025 - 10:30
Nov 9, 2026 - 01:06
 0  91
How Liveness and Readiness Probes Improve Reliability

Liveness probes restart a container that is stuck, and readiness probes stop sending traffic to a Pod that cannot serve it yet. Together they let Kubernetes heal hangs automatically, hold rollouts until new Pods are ready, and keep users away from broken replicas. Configured badly, the same probes can cause restart loops and cascading outages.

This post looks at reliability: what actually happens in the cluster when a probe passes or fails, and how to choose settings that make an app more stable rather than less. For a catalogue of every check type (HTTP, TCP, exec, gRPC, readiness gates and more) with examples, see our companion guide 10 Kubernetes Health Checks Every DevOps Must Use. For the bigger picture of how the kubelet, API server and controllers fit together, start with Kubernetes Architecture: 12 Components Explained.

Key takeaways

  • Readiness failure removes a Pod from Service endpoints. The container keeps running.
  • Liveness failure makes the kubelet kill and restart the container. Use it only for states the app cannot recover from by itself.
  • A startup probe holds off liveness and readiness checks until a slow-starting app is up.
  • Rolling updates wait for new Pods to become Ready, so a good readiness probe is what makes zero-downtime deploys work.
  • Never make a liveness probe depend on a database or another service. One outage downstream will restart every replica.
  • Default timings (10s period, 1s timeout, 3 failures) are a starting point. Tune them from measured startup and response times.

What each probe changes in the cluster

ProbeQuestion it asksOn failureReliability benefit
StartupHas the app finished starting?After failureThreshold failures the container is killed and handled by the restart policySlow starters are not killed by liveness during boot
LivenessIs the process still able to make progress?kubelet kills the container; restart policy decides what happens nextRecovers from deadlocks and hangs without a human
ReadinessCan this Pod serve requests right now?Pod condition Ready becomes false and the endpoint is marked not readyTraffic goes only to Pods that can handle it

The kubelet on each node runs the probes. That detail matters: probes test the container from its own node, not from the client's point of view. A probe can pass while users still fail because of DNS, a network policy or an ingress problem.

How readiness keeps traffic away from broken Pods

A Service does not send traffic to Pods directly. The EndpointSlice controller tracks which Pods match the Service selector and records whether each endpoint is ready. kube-proxy, or your CNI's replacement for it, programs the node so only ready endpoints receive new connections.

When a readiness probe fails failureThreshold times in a row, the sequence is:

  1. The kubelet sets the Pod's Ready condition to false and reports it to the API server.
  2. The EndpointSlice controller marks that endpoint as not ready.
  3. Nodes update their forwarding rules, and new connections stop going to the Pod.
  4. When the probe passes successThreshold times, the Pod returns to the pool.

Nothing restarts. This is the right reaction to temporary trouble such as a warming cache, a full request queue, or a dependency that is briefly unavailable. The Pod rests and comes back by itself.

Readiness has effects beyond Services. A Deployment counts only Ready Pods as available. A PodDisruptionBudget counts only healthy Pods when deciding whether a node drain may evict another one. If readiness lies, both of those protections lie too.

How liveness heals hung processes, and how it can hurt

Some failures do not crash the process. A thread pool deadlocks, an event loop blocks, or the JVM spends all its time in garbage collection. The process is alive, so the container never exits, but it serves nothing. A liveness probe turns that silent failure into a restart.

The same mechanism becomes dangerous when the probe checks the wrong thing. Consider a liveness endpoint that queries the database. The database has a short outage, every replica's liveness probe fails, and the kubelet restarts all of them at once. The database comes back to a stampede of cold-starting Pods, and some fail liveness again during startup. The result is a CrashLoopBackOff across the whole Deployment caused by a problem that readiness alone would have absorbed.

Rules that avoid this:

  • A liveness check should test only the process itself: can it answer a trivial request from its own code path?
  • Put dependency checks in readiness, or leave them out of probes entirely and rely on retries and circuit breakers in the app.
  • Give liveness more tolerance than readiness. A higher failureThreshold or periodSeconds means a short pause triggers removal from traffic before it triggers a restart.
  • If the app already crashes cleanly when something is wrong, you may not need a liveness probe at all. The restart policy handles exits.

Startup probes for slow starters

Before startup probes existed, teams set a long initialDelaySeconds on liveness to avoid killing an app during boot. That delay also applied after every restart and slowed detection of real hangs. A startup probe solves this cleanly. While it is running, liveness and readiness are disabled. Once it succeeds, it never runs again for that container, and the normal probes take over.

Size it from measured startup time: failureThreshold multiplied by periodSeconds should exceed the slowest realistic startup. For an app that takes up to 90 seconds, periodSeconds: 5 with failureThreshold: 24 gives two minutes.

Probes and zero-downtime rollouts

A rolling update replaces Pods in batches limited by maxSurge and maxUnavailable (both 25% by default). The Deployment controller creates new Pods, waits until they are Ready (and stay Ready for minReadySeconds, if set), and only then scales down old Pods. Readiness is the gate.

Probe setupRollout behaviour
No readiness probeA Pod is Ready as soon as its containers start. Traffic can hit it before the app listens, causing errors during every deploy.
Readiness that only checks the portBetter, but an app that listens before it loads config may still serve errors.
Readiness that checks the app can handle a real requestOld Pods keep serving until new ones can take over. A broken release never becomes Ready, and the rollout stalls instead of replacing healthy Pods.

That last row is a big reliability win. If a new image fails readiness, the rollout stops with the old ReplicaSet still serving. kubectl rollout status reports the stall, and progressDeadlineSeconds (600 by default) eventually marks the Deployment as failed so your pipeline can alert or roll back with kubectl rollout undo.

Shutdown: the other half of zero downtime

When a Pod is deleted, Kubernetes marks it as terminating and removes it from ready endpoints, while the kubelet sends SIGTERM to the containers. These happen in parallel, so for a moment some nodes may still route new connections to a Pod that is shutting down. Two habits close the gap:

  • Handle SIGTERM by finishing in-flight requests before exiting, within terminationGracePeriodSeconds (30 seconds by default).
  • Add a short preStop sleep so endpoint updates spread before the app stops accepting connections. Kubernetes 1.30 and later also support a built-in sleep action for preStop hooks.

A reliability-focused example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  minReadySeconds: 10
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: registry.example.com/orders-api:1.8.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 6
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]

Here /healthz answers from the process alone, while /ready also checks that config is loaded and the connection pool has capacity. Readiness reacts within about 10 seconds; liveness waits about a minute of consecutive failures before restarting. maxUnavailable: 0 keeps full capacity during rollouts.

Choosing probe timings

FieldDefaultHow to choose
initialDelaySeconds0Usually leave at 0 and use a startup probe instead
periodSeconds10Shorter for readiness, longer for liveness
timeoutSeconds1Above the endpoint's p99 latency under load
failureThreshold3Higher for liveness so pauses do not cause restarts
successThreshold1Must be 1 for liveness and startup; can be higher for readiness

A 1 second timeout is tight for apps under heavy load or with stop-the-world garbage collection. Measure, then set it.

Signals that probes are hurting reliability

  • Rising restart counts in kubectl get pods with no crash logs: liveness is probably too strict.
  • Unhealthy events in kubectl describe pod during traffic peaks: timeouts are too short for loaded endpoints.
  • All replicas going not-ready together: readiness depends on a shared dependency, so the Service has no endpoints at all. Consider whether serving degraded responses is better than serving nothing.
  • Errors only during deploys: missing or shallow readiness, or no graceful shutdown.

Alert on container restarts and on Deployments with fewer available replicas than desired. Our observability metrics guide lists the kube-state-metrics series for both. Probe failures are also a favourite interview topic; see Kubernetes scenario-based interview questions for practice.

Probes behind a service mesh or sidecar

Sidecars change what a probe sees. With Istio, for example, the sidecar intercepts traffic to the Pod, and mutual TLS would make plain HTTP probes from the kubelet fail. Istio handles this by rewriting HTTP probes so the kubelet calls the sidecar agent, which forwards the check to the app. If you run a mesh, read its probe documentation before assuming a probe failure means the app is broken.

Native sidecar containers (init containers with restartPolicy: Always) can carry their own startup, liveness and readiness probes. The Pod is Ready only when all containers that have readiness probes pass, so a slow log shipper or proxy can hold a Pod out of service. Give sidecar probes the same care as the main container's.

Test probe behaviour before production

Probe settings are guesses until you watch them under failure. A short test plan on a staging cluster catches most mistakes:

  1. Slow start. Add an artificial delay to startup and confirm the startup probe, not liveness, decides when the app is up. Watch kubectl get pods -w for unwanted restarts.
  2. Hang. Trigger a debug endpoint that blocks the request thread, or send SIGSTOP to the process from an ephemeral debug container. Liveness should restart the container after roughly periodSeconds times failureThreshold.
  3. Dependency outage. Block the database with a NetworkPolicy. Pods should go not-ready (if readiness checks it) but must not restart.
  4. Load. Run a load test and check probe latency in kubectl describe pod events. Timeouts under load are a sign timeoutSeconds is too tight.
  5. Bad release. Deploy an image whose readiness endpoint returns 500. The rollout should stall with old Pods still serving.
  6. Drain. Run kubectl drain on a node while sending traffic, and confirm no requests fail during eviction.
kubectl get pods -l app=orders-api -w
kubectl describe pod orders-api-7d9c6b5f4-abcde | sed -n '/Events/,$p'
kubectl get endpointslices -l kubernetes.io/service-name=orders-api -o wide
kubectl rollout status deployment/orders-api --timeout=5m

Four failure scenarios and what the probes do

A memory leak slowly makes the app unresponsive

Response times climb until the liveness endpoint times out. Readiness, which runs more often with a lower threshold, removes the Pod from traffic first. If the app does not recover, liveness restarts it. Users see little, but restart counts climb, which is your cue to fix the leak rather than rely on restarts.

A configuration error ships in a new release

New Pods start but cannot read a required secret, so /ready fails. With maxUnavailable: 0, no old Pods are removed. The rollout hits its progress deadline, the pipeline alerts, and you run kubectl rollout undo. Without a readiness check that loads config, the bad Pods would have received traffic.

The shared cache cluster goes down

If readiness checks the cache, every Pod goes not-ready at once and the Service has no endpoints, so users get connection errors instead of slower responses. If the app can serve from the database without the cache, leave the cache out of readiness and let the app degrade.

A node loses network connectivity

Probes run on the node, so the kubelet may still see the app as healthy. The control plane notices instead: the node stops reporting, it becomes NotReady, and after the eviction timeout its Pods are replaced elsewhere. Probes do not cover this case; node health and multi-zone spreading do.

Frequently asked questions

How do liveness and readiness probes improve reliability?

Liveness restarts containers that hang, so failures recover without a human. Readiness keeps traffic away from Pods that cannot serve and holds rollouts until new Pods are ready, which prevents errors during deploys.

Does a failed readiness probe restart the Pod?

No. It only marks the Pod not ready and removes it from Service endpoints. The container keeps running and rejoins when the probe passes again.

Should a liveness probe check the database?

No. If the database goes down, every replica fails liveness and restarts at the same time. Check dependencies in readiness or handle them in application code.

Can probes cause outages?

Yes. Strict liveness settings cause restart loops, and readiness checks tied to a shared dependency can remove every Pod from a Service at once.

Do I need a startup probe?

Use one when startup time is long or varies a lot, such as JVM apps or apps that load large data at boot. It prevents liveness from killing the container while it starts.

How do readiness probes help rolling updates?

The Deployment waits for new Pods to be Ready before removing old ones. A broken release never becomes Ready, so the rollout stalls while the old version keeps serving.

What happens if I set no probes at all?

Kubernetes treats the container as live while the process runs and ready as soon as it starts. Hangs go unnoticed and traffic can arrive before the app is ready.

Practise probes on a real cluster

Probe design, rollouts and troubleshooting are part of the hands-on labs in our CKAD training, run in our Kharadi, Pune classroom and live online across India. For deeper reading on workload types, see StatefulSet vs Deployment.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.