10 Hidden Kubernetes Logs You Should Check

Ten Kubernetes log sources beyond kubectl logs: previous containers, events, kubelet, containerd, kernel, control plane, etcd, audit logs and CoreDNS.

Dec 29, 2025 - 16:08
Nov 11, 2026 - 07:18
 0  89
10 Hidden Kubernetes Logs You Should Check

When kubectl logs shows nothing useful, the answer is usually in a log most people never open: the previous container's output, Kubernetes events, the kubelet and container runtime journals on the node, kernel messages, control plane component logs, etcd, the API server audit log, CoreDNS, or init and sidecar containers.

This guide lists ten of these log sources, where each one lives, the command to read it and the failures it explains. It assumes you know the basics of Pods and nodes; if not, start with 12 must-know Kubernetes architecture components. For tools that collect and search application logs at scale, see 12 Kubernetes logging tools for application insights.

Key takeaways

  • kubectl logs shows only the current container's stdout and stderr. A crash loop usually needs --previous.
  • Events explain scheduling, image pull and probe failures, but the API server keeps them for only one hour by default.
  • Node-level problems (Pod stuck in ContainerCreating, volume mount errors, OOM kills) appear in the kubelet journal, the container runtime journal and kernel messages.
  • Audit logs are the only record of who changed what through the API, and they are off until configured.
  • On managed services you cannot SSH into the control plane. Enable the provider's control plane log export before you need it.

The ten log sources at a glance

#Log sourceWhere it livesHow to read itExplains
1Previous container instanceNode, kept by the kubeletkubectl logs --previousCrashLoopBackOff causes
2Init and sidecar containersNodekubectl logs -c NAMEPods stuck in Init, proxy errors
3Kubernetes eventsAPI server (etcd), short-livedkubectl get events, kubectl eventsScheduling, image pull, probe failures
4Kubeletsystemd journal on each nodejournalctl -u kubeletContainerCreating, mount and CNI errors
5Container runtimesystemd journal on each nodejournalctl -u containerd, crictlSandbox and image errors
6KernelKernel ring buffer, journaljournalctl -k, dmesgOOM kills, disk and network driver errors
7API server, scheduler, controller managerStatic Pods or provider log servicekubectl logs -n kube-system, crictl logsAdmission errors, Pending Pods, stuck controllers
8etcdStatic Pod or provider-managedkubectl logs -n kube-system etcd-NODESlow API, leader elections, disk latency
9API server audit logFile or webhook backend you configureLog platform or file on control planeWho deleted or changed a resource
10CoreDNSPods in kube-systemkubectl logs -n kube-system -l k8s-app=kube-dnsName resolution failures

1. The previous container instance

When a container crashes and restarts, kubectl logs shows the new instance, which may have printed nothing yet. The crash reason is in the instance that just died.

kubectl logs payments-7d9c8b6f5-x2kqp --previous
kubectl logs payments-7d9c8b6f5-x2kqp -c app --previous --tail=100

Pair it with kubectl describe pod, which shows the last state, exit code and reason. Exit code 137 with reason OOMKilled means the container exceeded its memory limit. Exit code 1 usually points to an application error that the previous log will show. More crash patterns are in 15 common Kubernetes Pod failures and fixes.

2. Init containers and sidecars

A Pod stuck in Init:0/2 or Init:CrashLoopBackOff has an init container failing, and kubectl logs POD without -c will not show it (for multi-container Pods kubectl picks a default container or asks you to choose).

# List container names, including init containers
kubectl get pod web-0 -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}{"\n"}'

# Read the init container log
kubectl logs web-0 -c wait-for-db

# All containers at once, prefixed with the container name
kubectl logs web-0 --all-containers --prefix

Service mesh sidecars (for example Envoy in Istio) log connection resets, TLS failures and upstream timeouts that the application never sees. If an app reports "connection refused" but its own logs look clean, check the proxy container.

3. Kubernetes events

Events record what controllers and the kubelet did to objects: scheduling decisions, image pulls, probe failures, volume attach errors, evictions. They are the quickest explanation for a Pod stuck in Pending or ContainerCreating.

# Recent events in a namespace, sorted by time
kubectl get events -n payments --sort-by=.lastTimestamp

# Only warnings
kubectl get events -n payments --field-selector type=Warning

# Events for one object
kubectl events -n payments --for pod/payments-7d9c8b6f5-x2kqp

The catch: the API server keeps events for one hour by default (the --event-ttl flag). If you investigate the next morning, they are gone. Export events to your log platform with an event exporter or your observability agent so you can search them later.

4. The kubelet journal

The kubelet runs on every node and turns Pod specs into running containers. When a Pod sits in ContainerCreating, a volume will not mount, or the CNI plugin fails to set up networking, the kubelet logs the details.

# On the node (SSH or a node debug Pod)
journalctl -u kubelet --since "30 min ago"
journalctl -u kubelet -f | grep -i -E "error|failed"

# Without SSH: start a debug Pod on the node, then chroot into the host
kubectl debug node/worker-2 -it --image=busybox
chroot /host journalctl -u kubelet --since "30 min ago"

Common findings: failed to set up sandbox container ... network (CNI problem), MountVolume.SetUp failed (storage), certificate errors talking to the API server, and eviction messages when the node runs low on memory or disk. Node-level checks are also covered in 10 Kubernetes node maintenance tips.

5. The container runtime

Since dockershim was removed in Kubernetes 1.24, nodes run a CRI runtime such as containerd or CRI-O. The runtime logs image pull problems, sandbox creation failures and shim errors that the kubelet only summarises.

journalctl -u containerd --since "30 min ago"
# or, on CRI-O nodes
journalctl -u crio --since "30 min ago"

# Inspect containers directly through the CRI
crictl ps -a
crictl logs CONTAINER_ID
crictl inspectp POD_SANDBOX_ID

crictl also helps when the API server is down and kubectl cannot reach the cluster. The container log files themselves live under /var/log/pods/NAMESPACE_POD_UID/CONTAINER/ on the node, which is what log collectors such as Fluent Bit tail. See containerd vs Docker in Kubernetes for runtime background.

6. Kernel messages on the node

Some failures happen below Kubernetes. The kernel OOM killer can terminate processes when the node itself runs out of memory, disks can throw I/O errors, and network drivers can reset.

journalctl -k --since "1 hour ago"
journalctl -k | grep -i -E "oom|killed process"
dmesg -T | tail -50

A container hitting its own memory limit shows OOMKilled in kubectl describe. A node-wide OOM event may kill processes without a clear Pod status, so the kernel log is the place to confirm it. Repeated I/O errors here explain slow or failing persistent volumes before the storage layer reports anything.

7. Control plane component logs

The API server, scheduler and controller manager each write their own logs. On kubeadm clusters they run as static Pods in kube-system:

kubectl logs -n kube-system kube-apiserver-cp-1
kubectl logs -n kube-system kube-scheduler-cp-1
kubectl logs -n kube-system kube-controller-manager-cp-1

# If the API server is down, read the container directly on the control plane node
crictl ps -a | grep kube-apiserver
crictl logs CONTAINER_ID

What to look for:

  • API server: admission webhook timeouts (a broken webhook can block all Pod creation), certificate errors, etcd connection failures.
  • Scheduler: why Pods cannot be placed, beyond the summary in events (taints, affinity, resource fit).
  • Controller manager: stuck Deployments, failing volume provisioning, node lifecycle and garbage collection issues.

On EKS, GKE and AKS you cannot read these Pods. Each provider lets you export control plane logs to its log service (for example CloudWatch Logs for EKS), usually off by default for some log types. Turn them on during cluster setup.

8. etcd

When the whole cluster feels slow, with kubectl commands taking seconds, etcd is a prime suspect. etcd is very sensitive to disk latency.

kubectl logs -n kube-system etcd-cp-1 | grep -i -E "took too long|slow|leader"

Warnings such as apply request took too long and frequent leader elections suggest slow disks or an overloaded member. Check etcd's metrics alongside the log (WAL fsync and backend commit durations) and keep etcd on fast, dedicated storage. Managed services run etcd for you, so these logs are not available there.

9. The API server audit log

Every request to the API server can be recorded with who made it, from where, what it targeted and whether it was allowed. This is how you answer "who deleted the production namespace?" or "which service account keeps reading Secrets?"

Audit logging needs a policy file passed with --audit-policy-file and a backend such as --audit-log-path. Without a policy, nothing is recorded. A single audit event looks like this (trimmed):

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "stage": "ResponseComplete",
  "verb": "delete",
  "user": { "username": "[email protected]" },
  "sourceIPs": ["10.0.4.21"],
  "objectRef": { "resource": "deployments", "namespace": "payments", "name": "api" },
  "responseStatus": { "code": 200 }
}
# On a kubeadm control plane, find recent deletes in the payments namespace
sudo jq -c 'select(.verb=="delete" and .objectRef.namespace=="payments")
  | {time: .requestReceivedTimestamp, user: .user.username, obj: .objectRef}' \
  /var/log/kubernetes/audit/audit.log

Audit logs grow quickly, so log Secrets at Metadata level only and ship them off the node. Policy design is covered in 10 Kubernetes cluster hardening steps.

10. CoreDNS

Many "the service is down" reports are DNS failures. CoreDNS runs as Pods in kube-system (with the label k8s-app=kube-dns for compatibility).

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

# Test resolution from inside the cluster
kubectl run dnscheck --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default

By default CoreDNS logs errors but not every query. To see queries while troubleshooting, add the log plugin to the Corefile in the coredns ConfigMap, then remove it afterwards because it is noisy. Look for upstream timeouts, SERVFAIL responses and loop detection errors. 12 steps to debug networking in Kubernetes walks through the wider network checks.

Which log to open first

SymptomOpen firstThen
CrashLoopBackOffkubectl logs --previousdescribe pod (exit code), kernel log if OOM
PendingEventsScheduler log
ContainerCreating for minutesEventsKubelet journal, runtime journal
Init:ErrorInit container logEvents
ImagePullBackOffEventsRuntime journal
Intermittent 503s through a meshSidecar proxy logApplication log
Name resolution errorsCoreDNS logsNetwork policies, kubelet DNS config
kubectl very slowAPI server logetcd log and metrics
Pods cannot be created at allAPI server log (webhooks)Events on the ReplicaSet
A resource changed or disappearedAudit logController manager log

Make hidden logs visible before an incident

  • Collect node journals (kubelet, runtime, kernel) with your log agent, not only container stdout.
  • Export events so they outlive the one-hour default.
  • Enable audit logging with a sensible policy, or the provider's audit log export.
  • Enable control plane log export on managed clusters.
  • Keep a debug image and kubectl debug permissions ready for on-call engineers.

For the command-line side of troubleshooting, see top 15 Kubernetes troubleshooting commands.

Learn Kubernetes troubleshooting hands-on

Troubleshooting nodes, control plane components and networking is a large part of the CKA exam. Our CKA training in Pune practises these log sources on real clusters, in the Kharadi classroom and live online across India. Centralising logs is covered in the ELK Stack course.

Frequently asked questions

How do I see logs from a crashed Kubernetes container?

Run kubectl logs POD --previous, adding -c CONTAINER for multi-container Pods. It shows output from the instance that crashed rather than the one that just restarted.

How long are Kubernetes events kept?

One hour by default, set by the API server's --event-ttl flag. Export events to a log platform if you need them for later investigation.

Where are kubelet logs stored?

On systemd-based nodes the kubelet logs to the journal, so read them with journalctl -u kubelet on the node, or through a kubectl debug node session.

Where are container log files on a Kubernetes node?

Under /var/log/pods/, in a directory per namespace, Pod name and UID, with a subdirectory per container. Log collectors read these files.

How do I find out who deleted a Kubernetes resource?

Check the API server audit log for a delete verb on that resource. If audit logging was not enabled, the information was not recorded.

Can I read control plane logs on EKS, GKE or AKS?

Not by SSH. Each provider can export API server, audit, scheduler and controller manager logs to its log service, which you enable in the cluster settings.

How do I see CoreDNS queries?

Add the log plugin to the Corefile in the coredns ConfigMap in kube-system. CoreDNS then logs each query. Remove it after troubleshooting because it generates a lot of output.

What tool reads container logs when kubectl does not work?

Use crictl on the node. crictl ps -a lists containers, including control plane static Pods, and crictl logs shows their output even if the API server is down.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.