10 Kubernetes CrashLoopBackOff Fixes You Must Know
Fix Kubernetes CrashLoopBackOff: read previous logs and exit codes, then work through 10 fixes for probes, OOMKilled, config, permissions and commands.
CrashLoopBackOff means a container in your pod starts, exits, and is restarted by the kubelet again and again, with a growing wait between attempts. It is a symptom, not a cause. To fix it, read the previous container logs, check the exit code in kubectl describe pod, then work through the ten fixes below.
This guide is written as a diagnosis order. Start with the two commands that tell you which fix applies, then jump to the matching section. Commands were checked against the Kubernetes documentation for version 1.37, the current minor release in November 2026.
Key takeaways
- The kubelet restarts a crashed container with an exponential delay of 10s, 20s, 40s and so on, capped at 300 seconds (5 minutes). The timer resets after the container runs for 10 minutes without problems.
kubectl logs POD --previousshows the output of the crashed run. Plainkubectl logsoften shows an empty log for the new attempt.- The exit code and the
Reasonfield inkubectl describe podnarrow the cause in under a minute: 137 with OOMKilled is memory, 127 is a missing command, 1 is usually the application. - A liveness probe that fires before the app has finished starting is a common self-inflicted cause. A startup probe is the supported fix.
- A container whose main process finishes on purpose also loops, because the default
restartPolicyisAlways. kubectl debuglets you attach an ephemeral container or debug a copy of the pod when the image has no shell.
What CrashLoopBackOff actually means
When a container in a pod terminates, the kubelet applies the pod's restartPolicy. The values are Always, OnFailure and Never, and the default is Always. With Always, the kubelet restarts the container whether it exited with an error or finished cleanly. After each restart the kubelet waits longer: 10 seconds, 20, 40, and so on up to 300 seconds. Once a container has run for 10 minutes without a problem, the timer resets. While the kubelet is waiting, kubectl get pods shows CrashLoopBackOff in the STATUS column and the RESTARTS count keeps climbing.
So the status tells you the pattern (start, die, wait), not the reason. Your job is to find out why the process dies. Kubernetes also has feature gates that change these timings: ReduceDefaultCrashLoopBackOffDecay (alpha, off by default) shortens the delay, and KubeletCrashLoopBackOffMax (beta since v1.35) lets an admin lower the cap per node in the kubelet configuration. Unless your cluster sets them, plan around the standard timings.
Other pod states look similar and have different fixes. ImagePullBackOff means the image could not be pulled. CreateContainerConfigError means a referenced Secret or ConfigMap key is missing, so the container never started. Pending means the scheduler has not placed the pod. Our guide to 15 common Kubernetes pod failures and fixes covers those states, and this page stays on the crash loop.
Step zero: the two commands that point to the fix
Read the logs of the crashed run
kubectl logs my-app-5d8c9f7b6d-abcde --previous
# multi-container pod: name the container
kubectl logs my-app-5d8c9f7b6d-abcde -c app --previous
The --previous flag returns the log of the last terminated instance. Without it you may get the log of a container that has only just started and has nothing to say yet.
Read the exit code and the events
kubectl describe pod my-app-5d8c9f7b6d-abcde
Look at three places in the output. Under State and Last State, find Reason and Exit Code. Check Restart Count. Read the Events list at the bottom, where the kubelet records probe failures and kills. The page on debugging pods in the Kubernetes documentation explains these fields.
| Exit code | Usual meaning | Go to fix |
|---|---|---|
| 1 | Application error, uncaught exception, failed startup check | 1, 2, 7 |
| 0 | Process finished normally and was restarted anyway | 10 |
| 126 | Command found but not executable | 4, 8 |
| 127 | Command not found in the image | 4 |
| 137 | Killed with SIGKILL (128 + 9). With reason OOMKilled, memory limit hit | 3, 6 |
| 139 | Segmentation fault (128 + 11) | Application or image bug |
| 143 | Stopped with SIGTERM (128 + 15), often a failed liveness probe or a rollout | 6 |
Exit codes above 128 follow the Linux shell convention of 128 plus the signal number. The table is a starting point and not a verdict, so confirm with the logs and events.
The 10 fixes
Fix 1. Read the application error and fix the real bug
Most crash loops are plain application failures: a stack trace, a refused database connection, a port already in use, a syntax error in a config file. The previous logs show it. If the log is empty, the process may be dying before it logs anything, which points to fixes 4 and 5.
Container logs are limited when the process dies abruptly. Kubernetes can also read a short termination message. Set terminationMessagePolicy: FallbackToLogsOnError and the kubelet uses the last part of the log (up to 2048 bytes or 80 lines) as the message when the container exits with an error and wrote no message file:
containers:
- name: app
image: registry.example.com/shop/api:1.8.2
terminationMessagePolicy: FallbackToLogsOnError
Then read it without opening the logs:
kubectl get pod my-app-5d8c9f7b6d-abcde \
-o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"
Fix 2. Correct missing or wrong environment variables, ConfigMaps and Secrets
Applications that read required settings at startup exit with code 1 when a value is missing or malformed. Check that the names in the pod spec match what exists in the namespace:
kubectl get configmap app-config -o yaml
kubectl get secret db-credentials -o jsonpath='{.data}' | head -c 200
kubectl exec POD -- env | sort # only works if the container stays up long enough
If a referenced key does not exist, the pod usually shows CreateContainerConfigError instead. A crash loop in this area more often means the key exists but the value is wrong, for example a database host with a trailing space or a URL pointing at a service in another namespace. Secret values are base64 encoded, not encrypted, so decode them when you compare.
Fix 3. Raise memory limits or fix the memory use (OOMKilled)
If describe shows Reason: OOMKilled and exit code 137, the container went over its memory limit and the kernel killed it. Check the limit and the real use:
kubectl get pod my-app-5d8c9f7b6d-abcde -o jsonpath='{.spec.containers[*].resources}'
kubectl top pod my-app-5d8c9f7b6d-abcde --containers
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
Raising the limit is the quick fix. The lasting fix depends on the runtime. A JVM sized for the whole node needs an explicit heap setting that fits inside the container limit, and a Node.js or Python service with a leak needs the leak fixed. Setting a memory request equal to the limit also makes scheduling decisions predictable. See our guide to Kubernetes resource optimization techniques for sizing from real usage.
Fix 4. Repair the command, arguments and entrypoint
Exit codes 126 and 127, or a log line such as exec format error, mean the container could not run the command. The usual causes are a wrong command or args override in the manifest, a script without the executable bit, a missing interpreter line, Windows line endings in a shell script, or an image built for a different CPU architecture than the node.
containers:
- name: app
image: registry.example.com/shop/api:1.8.2
command: ["/app/server"]
args: ["--port=8080", "--config=/etc/app/config.yaml"]
Remember that command in a pod replaces the image ENTRYPOINT and args replaces CMD. Overriding one and forgetting the other is a classic mistake. Run the image locally with the same arguments to confirm it starts:
docker run --rm registry.example.com/shop/api:1.8.2 /app/server --port=8080
Fix 5. Make the dependencies available, or let the app wait for them
An API that exits because the database is not reachable yet will crash-loop until the database comes up. The best fix is in the application: retry the connection with a short delay instead of exiting. If you cannot change the code, an init container can block the main container until a dependency answers:
initContainers:
- name: wait-for-db
image: busybox:1.36
command: ["sh", "-c", "until nc -z db-service 5432; do echo waiting; sleep 2; done"]
Init containers run to completion, in order, before app containers start. If the init container itself fails, the pod shows Init:CrashLoopBackOff and you read its logs with kubectl logs POD -c wait-for-db. Also test that DNS and network policies allow the connection, because a blocked port looks the same as a dead service from inside the pod.
Fix 6. Tune liveness probes and add a startup probe
A liveness probe tells the kubelet to restart a container that is stuck. If it starts checking before the application is ready, the kubelet kills a healthy but slow container, and it restarts into the same trap. The events section of describe shows lines such as Liveness probe failed followed by Killing.
The Kubernetes documentation recommends a startup probe for slow starters. While a startup probe has not yet succeeded, the liveness and readiness probes are held back. The documentation's own example allows a maximum of 5 minutes to start by combining failureThreshold: 30 with periodSeconds: 10:
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 3
timeoutSeconds: 2
The defaults are worth knowing: periodSeconds is 10, timeoutSeconds is 1, failureThreshold is 3 and successThreshold is 1. A one second timeout is tight for an endpoint that touches a database, so raise timeoutSeconds if you see timeouts under load. Keep the liveness check cheap and local. It should not call downstream services, or one slow dependency will restart every replica at once. Our posts on liveness and readiness probes and Kubernetes health checks go deeper.
Fix 7. Fix the port, host binding and health endpoint
A container that runs fine but fails its probe will still be restarted. Common causes: the app listens on 127.0.0.1 inside the container instead of 0.0.0.0, so the kubelet cannot reach it; the probe path or port does not match what the app serves; or the app starts on a different port than containerPort suggests. Confirm from inside the pod network:
kubectl port-forward pod/my-app-5d8c9f7b6d-abcde 8080:8080
curl -i http://localhost:8080/healthz
For HTTP probes, any status code from 200 up to but not including 400 counts as success. A health endpoint that returns 401 because it sits behind authentication will fail the probe every time.
Fix 8. Resolve file permissions and security context conflicts
Hardened clusters enforce settings such as runAsNonRoot and readOnlyRootFilesystem. An image that writes to its own filesystem, or that needs to bind a port below 1024 as a non-root user, will exit with a permission error. Read the log for permission denied or read-only file system, then fix the cause instead of loosening the policy:
securityContext:
runAsNonRoot: true
runAsUser: 10001
readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
Give the app a writable emptyDir for temporary files and have it listen on a high port such as 8080. For mounted volumes, fsGroup in the pod security context sets group ownership so a non-root process can write. Our post on Kubernetes security best practices explains why these restrictions are worth keeping.
Fix 9. Check volumes and mounts
Mount problems show up two ways. A missing PersistentVolumeClaim or ConfigMap keeps the pod from starting at all, with the reason in the events. The quieter one is a mount that hides files from the image: mounting a volume at /app/config replaces everything the image placed in that directory. If the app then cannot find its default files, it exits. Use subPath to mount a single file, or mount at a new path:
volumeMounts:
- name: app-config
mountPath: /app/config/settings.yaml
subPath: settings.yaml
volumes:
- name: app-config
configMap:
name: app-config
One limit to remember: a ConfigMap mounted with subPath does not receive live updates when the ConfigMap changes.
Fix 10. Stop processes that exit on purpose from running as services
Exit code 0 in a crash loop is the giveaway. The container did its job and ended, and because restartPolicy is Always the kubelet starts it again. Common examples: a shell script that finishes, a command of ["echo", "hello"], or an application that daemonizes into the background so PID 1 returns immediately. Containers need a foreground main process.
If the task is a one-time or scheduled job, use a Job or CronJob. Pods created by a Job use restartPolicy: Never or OnFailure, and Always is not allowed there:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
spec:
backoffLimit: 4
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/shop/migrate:1.8.2
When there is no shell: debug the crashing pod
Many production images are distroless or minimal, so kubectl exec fails. Two kubectl debug approaches handle that. An ephemeral container attaches a debugging image to the running pod:
kubectl debug -it my-app-5d8c9f7b6d-abcde --image=busybox:1.28 --target=app
Adding an ephemeral container needs permission on the pods/ephemeralcontainers subresource, which cluster RBAC may not grant by default. The second option copies the pod so you can change its command without touching the original:
kubectl debug my-app-5d8c9f7b6d-abcde -it --image=ubuntu --share-processes --copy-to=my-app-debug
The copy shares the process namespace, so you can look at the crashing process, and you can run it by hand with the arguments from the manifest. Delete the debug copy when you are finished. Our post on Kubernetes pod troubleshooting techniques and the list of troubleshooting commands add more tools.
Quick reference: symptom to fix
| What you see | Likely cause | Fix |
|---|---|---|
| Reason OOMKilled, exit 137 | Memory limit lower than real use | 3 |
| Stack trace or connection refused in previous logs | Application bug or missing dependency | 1, 5 |
| Log complains about a missing variable or key | Wrong ConfigMap or Secret data | 2 |
| Exit 126 or 127, or exec format error | Bad command, permissions or image architecture | 4 |
| Events show Liveness probe failed, then Killing | Probe too early or too strict | 6, 7 |
| permission denied, read-only file system | Security context versus app needs | 8 |
| Config files missing after a mount | Volume hides image files | 9 |
| Exit 0 with restarts | Process ends by design | 10 |
Prevent crash loops before they reach production
- Set realistic memory and CPU requests and limits from measured use, not guesses.
- Add a startup probe to every slow starting service, and keep liveness probes cheap.
- Log to standard output and standard error, and fail fast with a clear message when configuration is missing.
- Test the image with the exact command, arguments and security context you deploy, in a staging namespace.
- Alert on rising restart counts and on
OOMKilledterminations instead of waiting for a pod to sit in backoff. - Roll out with a rolling update and sensible readiness checks so a bad image never replaces all healthy pods. See Kubernetes rollout strategies.
If you want to practise these failures on a real cluster with an instructor, our CKA training in Pune (classroom in Kharadi, and live online across India) includes troubleshooting labs, and the CKAD course covers probes, resources and multi-container pods from the developer side.
Frequently asked questions
How long does CrashLoopBackOff last?
It lasts until the container stops crashing. The kubelet waits 10 seconds before the first retry, then 20, 40 and so on, up to a maximum of 5 minutes between attempts. If the container then runs for 10 minutes without a problem, the delay resets.
How do I see logs from a pod that keeps crashing?
Run kubectl logs POD --previous, adding -c CONTAINER for multi-container pods. This returns the output of the last terminated container instance, which is where the error message usually is. Plain kubectl logs shows the current attempt, which may be empty.
What does exit code 137 mean in Kubernetes?
It means the process received SIGKILL (128 plus 9). With Reason: OOMKilled in kubectl describe pod, the container exceeded its memory limit. Without that reason, 137 can also follow a liveness failure that ended in a forced kill after the grace period.
Can a liveness probe cause CrashLoopBackOff?
Yes. If the probe fails before the app is ready, the kubelet restarts a healthy container repeatedly. Add a startup probe with enough failureThreshold times periodSeconds to cover the slowest realistic start, and raise timeoutSeconds if the endpoint is slow under load.
Why does my pod crash only in Kubernetes and not in Docker?
Differences usually come from environment variables, mounted ConfigMaps and Secrets, memory limits, a non-root security context, a read-only filesystem, or the network. Run the image locally with the same user, limits and arguments to reproduce it, or use kubectl debug --copy-to to experiment in the cluster.
Is CrashLoopBackOff the same as ImagePullBackOff?
No. ImagePullBackOff means Kubernetes cannot download the image, so the container never starts. CrashLoopBackOff means the container does start and then exits. The first is fixed with the image name, tag or registry credentials, and the second needs the log and exit code.
How do I debug a container that has no shell?
Use kubectl debug -it POD --image=busybox:1.28 --target=CONTAINER to attach an ephemeral container, if your RBAC allows it. Or run kubectl debug POD -it --image=ubuntu --share-processes --copy-to=NAME to work on a copy of the pod.
Should I delete the pod to fix a crash loop?
Deleting a pod managed by a Deployment only creates a fresh pod with the same spec, and it will crash again if the cause is unchanged. Fix the manifest, image or configuration first, then apply the change so a new ReplicaSet rolls out.
References
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0