Application Troubleshooting

TL;DR

• Application-level failures collapse into a small set of recognizable pod states: Pending, ImagePullBackOff, CrashLoopBackOff, CreateContainerConfigError, and stuck Terminating.
• The diagnostic order is fixed and should not be skipped: describe (events) → logs (including --previous) → exec. Events usually name the root cause directly in plain English.
• Exit code 137 = SIGKILL, almost always OOMKilled. Exit code 143 = SIGTERM, typically an incomplete graceful shutdown.
• A Pending pod with “Insufficient cpu” does not mean the cluster is out of CPU — it means no single node individually has enough free allocatable capacity, since Kubernetes never splits one pod’s resource request across multiple nodes.
• Biggest trade-off: jumping straight to kubectl exec feels fast but frequently fails outright on pods with no running container yet (CrashLoopBackOff, Pending), wasting more time than reading events first would have.

Core Concepts

The Diagnostic Sequence

kubectl get pods -o wide                 # 1. Current state
kubectl describe pod <pod>                # 2. Events: what happened, and why
kubectl logs <pod> [--previous]           # 3. What the application itself reported
kubectl exec -it <pod> -- /bin/sh         # 4. Direct inspection inside the container

This order matters because each step is progressively more expensive and depends on the prior step’s output existing at all. describe surfaces scheduler and kubelet events before any container logic runs. logs --previous is only meaningful once a crash has actually occurred. exec requires a container process currently running — it is unavailable for a large fraction of failure states, which is why it belongs last, not first.

Common Pod States and Root Causes

CrashLoopBackOff

kubectl logs <pod> --previous     # what the app printed immediately before dying
kubectl describe pod <pod>        # "Last State: Terminated" reason and exit code

Exit code reference:

• 0 — clean exit, but the container is not meant to exit at all if its main process terminated; check the command/entrypoint for premature completion.
• 1 — generic application error; inspect logs.
• 137 — SIGKILL. Almost always OOMKilled — confirm via the OOMKilled reason in the container status, then check resource limits.
• 143 — SIGTERM. Typically a graceful shutdown that did not finish in time; check terminationGracePeriodSeconds.

A container repeatedly restarting despite passing readiness probes is usually a liveness-probe failure independent of the application’s actual health — the probe configuration itself (timeout too short, wrong path/port) is often the real defect, not the application.

Pending Pods

kubectl describe pod <pod> | grep -A5 Events

Exact event messages and their meaning:

• "Insufficient cpu" / "Insufficient memory" — no node has room; compare requests against actual per-node allocatable capacity, or trigger the Cluster Autoscaler.
• "node(s) had taints that the pod didn't tolerate" — missing toleration.
• "no persistent volumes available for this claim" — PVC cannot bind; check StorageClass/provisioner.
• "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity" — affinity rule too strict for the current set of node labels.

The scheduler evaluates feasibility per node, never in aggregate. A cluster with abundant total free CPU can still fail to schedule a pod if every individual node’s leftover allocatable CPU falls short of the pod’s request — Kubernetes has no mechanism to fragment a single pod’s resource request across nodes.

CreateContainerConfigError

Triggered when the container spec references a ConfigMap or Secret key via envFrom or valueFrom that does not exist in the referenced object. Unlike a missing ConfigMap/Secret entirely (which produces a different, related error), this specifically means the object exists but the named key inside it does not — almost always a typo.

Stuck Terminating

A pod stuck in Terminating past its grace period usually means one of two things: a finalizer registered on the object is blocking garbage collection until some external controller acknowledges cleanup, or the node hosting the pod is unreachable so the kubelet can never report the pod as fully torn down. Distinguish these by checking kubectl get node for the hosting node’s status before assuming a finalizer issue.

Command / Configuration Reference

Check exact exit code and termination reason:

kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Run a temporary debug container attached to a running (possibly minimal/distroless) pod:

kubectl debug -it my-pod --image=busybox --target=my-container

Confirm an image pull secret is attached and correctly referenced:

kubectl get pod my-pod -o jsonpath='{.spec.imagePullSecrets}'

Force-delete a pod stuck in Terminating due to finalizers (use cautiously — can leak external resources the finalizer was meant to clean up):

kubectl patch pod my-pod -p '{"metadata":{"finalizers":null}}'

Get the events tied to a specific pod without wading through full describe output:

kubectl get events --field-selector involvedObject.name=my-pod --sort-by='.lastTimestamp'

CrashLoopBackOff vs ImagePullBackOff vs Pending vs OOMKilled

Common Pitfalls

• Pitfall: Running kubectl exec immediately on a struggling pod. Why: Many failure states (CrashLoopBackOff, Pending, ImagePullBackOff) have no running container to exec into. Fix: Read kubectl describe pod events and kubectl logs --previous first; they resolve the majority of cases without a shell.
• Pitfall: Concluding OOM did not happen because kubectl top pod currently shows memory well under the limit. Why: kubectl top reflects a live/recent snapshot, not the historical peak at the moment of the kill. Fix: Check exit code 137 and the OOMKilled reason directly, and correlate with historical metrics (Prometheus/Grafana) around the crash timestamp.
• Pitfall: Assuming “Insufficient cpu” means the cluster is out of capacity. Why: The scheduler evaluates per-node allocatable resources, not cluster-wide totals. Fix: Compare the pod’s request against each node’s actual free allocatable CPU/memory, not the sum across all nodes.
• Pitfall: Treating CreateContainerConfigError the same as a missing ConfigMap/Secret. Why: The object exists but a specific referenced key inside it is absent — a narrower, more specific misconfiguration. Fix: Diff the envFrom/valueFrom key name against the actual keys in the ConfigMap/Secret.
• Pitfall: Force-deleting a Terminating pod by wiping finalizers as a default reflex. Why: Finalizers often exist to clean up external resources (load balancer entries, storage attachments); bypassing them can leak those resources. Fix: Investigate why the finalizer’s controller has not acknowledged cleanup before removing it.

Interview Questions

• A pod shows CrashLoopBackOff with exit code 137, but kubectl top pod shows memory well under the limit right now — how do you reconcile this? — Tests understanding that kubectl top is a live snapshot, not historical peak usage, and that exit code interpretation should not be second-guessed by current-state metrics.
• A pod is Pending with “0/5 nodes are available: 5 Insufficient cpu” despite the cluster having ample total free CPU — what’s going on? — Tests understanding of per-node scheduling feasibility versus cluster-wide aggregate capacity.
• Walk through the exact sequence of commands to diagnose a CrashLoopBackOff pod from scratch. — Tests whether the candidate follows a disciplined describe-then-logs-then-exec order rather than guessing.
• What is the difference between CreateContainerConfigError and ImagePullBackOff? — Tests precise knowledge of failure-state taxonomy and where in the pod lifecycle each occurs.
• A pod is stuck Terminating well past its grace period — what are the two likely causes and how do you distinguish them? — Tests knowledge of finalizers versus unreachable-node scenarios and the safe way to unblock deletion.