10 Hidden Kubernetes Logs You Should Check
Ten Kubernetes log sources beyond kubectl logs: previous containers, events, kubelet, containerd, kernel, control plane, etcd, audit logs and CoreDNS.
When kubectl logs shows nothing useful, the answer is usually in a log most people never open: the previous container's output, Kubernetes events, the kubelet and container runtime journals on the node, kernel messages, control plane component logs, etcd, the API server audit log, CoreDNS, or init and sidecar containers.
This guide lists ten of these log sources, where each one lives, the command to read it and the failures it explains. It assumes you know the basics of Pods and nodes; if not, start with 12 must-know Kubernetes architecture components. For tools that collect and search application logs at scale, see 12 Kubernetes logging tools for application insights.
Key takeaways
kubectl logsshows only the current container's stdout and stderr. A crash loop usually needs--previous.- Events explain scheduling, image pull and probe failures, but the API server keeps them for only one hour by default.
- Node-level problems (Pod stuck in ContainerCreating, volume mount errors, OOM kills) appear in the kubelet journal, the container runtime journal and kernel messages.
- Audit logs are the only record of who changed what through the API, and they are off until configured.
- On managed services you cannot SSH into the control plane. Enable the provider's control plane log export before you need it.
The ten log sources at a glance
| # | Log source | Where it lives | How to read it | Explains |
|---|---|---|---|---|
| 1 | Previous container instance | Node, kept by the kubelet | kubectl logs --previous | CrashLoopBackOff causes |
| 2 | Init and sidecar containers | Node | kubectl logs -c NAME | Pods stuck in Init, proxy errors |
| 3 | Kubernetes events | API server (etcd), short-lived | kubectl get events, kubectl events | Scheduling, image pull, probe failures |
| 4 | Kubelet | systemd journal on each node | journalctl -u kubelet | ContainerCreating, mount and CNI errors |
| 5 | Container runtime | systemd journal on each node | journalctl -u containerd, crictl | Sandbox and image errors |
| 6 | Kernel | Kernel ring buffer, journal | journalctl -k, dmesg | OOM kills, disk and network driver errors |
| 7 | API server, scheduler, controller manager | Static Pods or provider log service | kubectl logs -n kube-system, crictl logs | Admission errors, Pending Pods, stuck controllers |
| 8 | etcd | Static Pod or provider-managed | kubectl logs -n kube-system etcd-NODE | Slow API, leader elections, disk latency |
| 9 | API server audit log | File or webhook backend you configure | Log platform or file on control plane | Who deleted or changed a resource |
| 10 | CoreDNS | Pods in kube-system | kubectl logs -n kube-system -l k8s-app=kube-dns | Name resolution failures |
1. The previous container instance
When a container crashes and restarts, kubectl logs shows the new instance, which may have printed nothing yet. The crash reason is in the instance that just died.
kubectl logs payments-7d9c8b6f5-x2kqp --previous
kubectl logs payments-7d9c8b6f5-x2kqp -c app --previous --tail=100
Pair it with kubectl describe pod, which shows the last state, exit code and reason. Exit code 137 with reason OOMKilled means the container exceeded its memory limit. Exit code 1 usually points to an application error that the previous log will show. More crash patterns are in 15 common Kubernetes Pod failures and fixes.
2. Init containers and sidecars
A Pod stuck in Init:0/2 or Init:CrashLoopBackOff has an init container failing, and kubectl logs POD without -c will not show it (for multi-container Pods kubectl picks a default container or asks you to choose).
# List container names, including init containers
kubectl get pod web-0 -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}{"\n"}'
# Read the init container log
kubectl logs web-0 -c wait-for-db
# All containers at once, prefixed with the container name
kubectl logs web-0 --all-containers --prefix
Service mesh sidecars (for example Envoy in Istio) log connection resets, TLS failures and upstream timeouts that the application never sees. If an app reports "connection refused" but its own logs look clean, check the proxy container.
3. Kubernetes events
Events record what controllers and the kubelet did to objects: scheduling decisions, image pulls, probe failures, volume attach errors, evictions. They are the quickest explanation for a Pod stuck in Pending or ContainerCreating.
# Recent events in a namespace, sorted by time
kubectl get events -n payments --sort-by=.lastTimestamp
# Only warnings
kubectl get events -n payments --field-selector type=Warning
# Events for one object
kubectl events -n payments --for pod/payments-7d9c8b6f5-x2kqp
The catch: the API server keeps events for one hour by default (the --event-ttl flag). If you investigate the next morning, they are gone. Export events to your log platform with an event exporter or your observability agent so you can search them later.
4. The kubelet journal
The kubelet runs on every node and turns Pod specs into running containers. When a Pod sits in ContainerCreating, a volume will not mount, or the CNI plugin fails to set up networking, the kubelet logs the details.
# On the node (SSH or a node debug Pod)
journalctl -u kubelet --since "30 min ago"
journalctl -u kubelet -f | grep -i -E "error|failed"
# Without SSH: start a debug Pod on the node, then chroot into the host
kubectl debug node/worker-2 -it --image=busybox
chroot /host journalctl -u kubelet --since "30 min ago"
Common findings: failed to set up sandbox container ... network (CNI problem), MountVolume.SetUp failed (storage), certificate errors talking to the API server, and eviction messages when the node runs low on memory or disk. Node-level checks are also covered in 10 Kubernetes node maintenance tips.
5. The container runtime
Since dockershim was removed in Kubernetes 1.24, nodes run a CRI runtime such as containerd or CRI-O. The runtime logs image pull problems, sandbox creation failures and shim errors that the kubelet only summarises.
journalctl -u containerd --since "30 min ago"
# or, on CRI-O nodes
journalctl -u crio --since "30 min ago"
# Inspect containers directly through the CRI
crictl ps -a
crictl logs CONTAINER_ID
crictl inspectp POD_SANDBOX_ID
crictl also helps when the API server is down and kubectl cannot reach the cluster. The container log files themselves live under /var/log/pods/NAMESPACE_POD_UID/CONTAINER/ on the node, which is what log collectors such as Fluent Bit tail. See containerd vs Docker in Kubernetes for runtime background.
6. Kernel messages on the node
Some failures happen below Kubernetes. The kernel OOM killer can terminate processes when the node itself runs out of memory, disks can throw I/O errors, and network drivers can reset.
journalctl -k --since "1 hour ago"
journalctl -k | grep -i -E "oom|killed process"
dmesg -T | tail -50
A container hitting its own memory limit shows OOMKilled in kubectl describe. A node-wide OOM event may kill processes without a clear Pod status, so the kernel log is the place to confirm it. Repeated I/O errors here explain slow or failing persistent volumes before the storage layer reports anything.
7. Control plane component logs
The API server, scheduler and controller manager each write their own logs. On kubeadm clusters they run as static Pods in kube-system:
kubectl logs -n kube-system kube-apiserver-cp-1
kubectl logs -n kube-system kube-scheduler-cp-1
kubectl logs -n kube-system kube-controller-manager-cp-1
# If the API server is down, read the container directly on the control plane node
crictl ps -a | grep kube-apiserver
crictl logs CONTAINER_ID
What to look for:
- API server: admission webhook timeouts (a broken webhook can block all Pod creation), certificate errors, etcd connection failures.
- Scheduler: why Pods cannot be placed, beyond the summary in events (taints, affinity, resource fit).
- Controller manager: stuck Deployments, failing volume provisioning, node lifecycle and garbage collection issues.
On EKS, GKE and AKS you cannot read these Pods. Each provider lets you export control plane logs to its log service (for example CloudWatch Logs for EKS), usually off by default for some log types. Turn them on during cluster setup.
8. etcd
When the whole cluster feels slow, with kubectl commands taking seconds, etcd is a prime suspect. etcd is very sensitive to disk latency.
kubectl logs -n kube-system etcd-cp-1 | grep -i -E "took too long|slow|leader"
Warnings such as apply request took too long and frequent leader elections suggest slow disks or an overloaded member. Check etcd's metrics alongside the log (WAL fsync and backend commit durations) and keep etcd on fast, dedicated storage. Managed services run etcd for you, so these logs are not available there.
9. The API server audit log
Every request to the API server can be recorded with who made it, from where, what it targeted and whether it was allowed. This is how you answer "who deleted the production namespace?" or "which service account keeps reading Secrets?"
Audit logging needs a policy file passed with --audit-policy-file and a backend such as --audit-log-path. Without a policy, nothing is recorded. A single audit event looks like this (trimmed):
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"stage": "ResponseComplete",
"verb": "delete",
"user": { "username": "[email protected]" },
"sourceIPs": ["10.0.4.21"],
"objectRef": { "resource": "deployments", "namespace": "payments", "name": "api" },
"responseStatus": { "code": 200 }
}
# On a kubeadm control plane, find recent deletes in the payments namespace
sudo jq -c 'select(.verb=="delete" and .objectRef.namespace=="payments")
| {time: .requestReceivedTimestamp, user: .user.username, obj: .objectRef}' \
/var/log/kubernetes/audit/audit.log
Audit logs grow quickly, so log Secrets at Metadata level only and ship them off the node. Policy design is covered in 10 Kubernetes cluster hardening steps.
10. CoreDNS
Many "the service is down" reports are DNS failures. CoreDNS runs as Pods in kube-system (with the label k8s-app=kube-dns for compatibility).
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
# Test resolution from inside the cluster
kubectl run dnscheck --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
By default CoreDNS logs errors but not every query. To see queries while troubleshooting, add the log plugin to the Corefile in the coredns ConfigMap, then remove it afterwards because it is noisy. Look for upstream timeouts, SERVFAIL responses and loop detection errors. 12 steps to debug networking in Kubernetes walks through the wider network checks.
Which log to open first
| Symptom | Open first | Then |
|---|---|---|
| CrashLoopBackOff | kubectl logs --previous | describe pod (exit code), kernel log if OOM |
| Pending | Events | Scheduler log |
| ContainerCreating for minutes | Events | Kubelet journal, runtime journal |
| Init:Error | Init container log | Events |
| ImagePullBackOff | Events | Runtime journal |
| Intermittent 503s through a mesh | Sidecar proxy log | Application log |
| Name resolution errors | CoreDNS logs | Network policies, kubelet DNS config |
| kubectl very slow | API server log | etcd log and metrics |
| Pods cannot be created at all | API server log (webhooks) | Events on the ReplicaSet |
| A resource changed or disappeared | Audit log | Controller manager log |
Make hidden logs visible before an incident
- Collect node journals (kubelet, runtime, kernel) with your log agent, not only container stdout.
- Export events so they outlive the one-hour default.
- Enable audit logging with a sensible policy, or the provider's audit log export.
- Enable control plane log export on managed clusters.
- Keep a debug image and
kubectl debugpermissions ready for on-call engineers.
For the command-line side of troubleshooting, see top 15 Kubernetes troubleshooting commands.
Learn Kubernetes troubleshooting hands-on
Troubleshooting nodes, control plane components and networking is a large part of the CKA exam. Our CKA training in Pune practises these log sources on real clusters, in the Kharadi classroom and live online across India. Centralising logs is covered in the ELK Stack course.
Frequently asked questions
How do I see logs from a crashed Kubernetes container?
Run kubectl logs POD --previous, adding -c CONTAINER for multi-container Pods. It shows output from the instance that crashed rather than the one that just restarted.
How long are Kubernetes events kept?
One hour by default, set by the API server's --event-ttl flag. Export events to a log platform if you need them for later investigation.
Where are kubelet logs stored?
On systemd-based nodes the kubelet logs to the journal, so read them with journalctl -u kubelet on the node, or through a kubectl debug node session.
Where are container log files on a Kubernetes node?
Under /var/log/pods/, in a directory per namespace, Pod name and UID, with a subdirectory per container. Log collectors read these files.
How do I find out who deleted a Kubernetes resource?
Check the API server audit log for a delete verb on that resource. If audit logging was not enabled, the information was not recorded.
Can I read control plane logs on EKS, GKE or AKS?
Not by SSH. Each provider can export API server, audit, scheduler and controller manager logs to its log service, which you enable in the cluster settings.
How do I see CoreDNS queries?
Add the log plugin to the Corefile in the coredns ConfigMap in kube-system. CoreDNS then logs each query. Remove it after troubleshooting because it generates a lot of output.
What tool reads container logs when kubectl does not work?
Use crictl on the node. crictl ps -a lists containers, including control plane static Pods, and crictl logs shows their output even if the API server is down.
References
- Kubernetes: logging architecture
- Kubernetes: system logs
- Kubernetes: troubleshooting clusters
- Kubernetes: auditing
- kube-apiserver reference (--event-ttl)
- Kubernetes: debugging nodes with crictl
- Kubernetes: debugging nodes with kubectl
- Kubernetes: debugging DNS resolution
- CoreDNS log plugin
- etcd FAQ (slow requests and disk latency)
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0