Worker Node Troubleshooting

TL;DR

• Node NotReady usually stems from: kubelet crashed, container runtime crashed, networking broken, or severe resource pressure
• Pods stuck in ContainerCreating: check container runtime health (crictl info), then CNI per-node component (DaemonSet)
• Node evicts pods under memory pressure in QoS order: BestEffort first, then Burstable, Guaranteed last
• Cgroup driver mismatch (systemd vs cgroupfs) between kubelet and container runtime causes crash-loops with cgroup errors post-upgrade
• NetworkUnavailable: True on new node is normal for 30 seconds while CNI DaemonSet pod initializes; not actionable initially
• Test container runtime directly: crictl info, crictl ps; independent of Kubernetes API

Core Concepts

Node Conditions

Ready: Node is healthy; kubelet communicating, disk/memory/PID not under pressure, networking operational.

NotReady: Node is unhealthy; kubelet not communicating, disk/memory/PID pressure, or networking failed.

MemoryPressure: Node memory approaching limit; kubelet evicts pods.

DiskPressure: Node disk full or near-full; kubelet evicts pods, stops accepting new pods.

PIDPressure: Process ID space exhausted; kubelet unable to spawn new containers.

NetworkUnavailable: Networking not yet initialized (normally for first 30 seconds on new node while CNI pod starts).

QoS-Based Pod Eviction

Under resource pressure, kubelet evicts pods in order:

1. BestEffort (no requests/limits) — evicted first
2. Burstable (requests < limits) — evicted second
3. Guaranteed (requests == limits) — evicted last

Container Runtime vs. Kubernetes

Container runtime (containerd, CRI-O, Docker) is independent of Kubernetes. kubelet communicates with runtime via CRI socket. If runtime crashes, kubelet cannot create/manage containers, but runtime itself is a separate process.

Cgroup Drivers

kubelet and container runtime must use the same cgroup driver:

• systemd: Modern, recommended
• cgroupfs: Legacy

Mismatch causes crash-loops with “cgroup” errors. Both must be configured at startup.

NetworkUnavailable Taint

New nodes start with NetworkUnavailable: True. CNI DaemonSet pod must start and initialize networking. Once ready, condition clears. Pods with toleration for this taint can start early (usually just kube-system pods).

Comparison Table

Condition	Cause	Recovery
NotReady	kubelet/runtime crashed, networking broken	SSH to node, restart service, check logs
MemoryPressure	RAM approaching limit	Add memory, evict pods, or scale up
DiskPressure	Disk full	Free space, clean up logs, or replace node
PIDPressure	Process ID exhaustion	Restart node, investigate zombie processes
NetworkUnavailable	CNI not initialized	Wait 30 seconds; check CNI pod status

Command Reference

Diagnose Node Status

# List nodes with conditions
kubectl get nodes -o wide

# Detailed node conditions
kubectl describe node <node-name>
# Look for: NotReady, MemoryPressure, DiskPressure, NetworkUnavailable

# Check kubelet logs (requires SSH)
ssh user@node
sudo journalctl -u kubelet -n 100
sudo journalctl -u kubelet --follow

# Check if kubelet is running
sudo systemctl status kubelet

Access Container Runtime Directly

# SSH to node, then:

# Check container runtime health
crictl info

# List running containers
crictl ps

# List all containers (including exited)
crictl ps --all

# Inspect container
crictl inspect <container-id>

# View container logs
crictl logs <container-id>

Check CNI Status

# List CNI DaemonSet pods
kubectl get pods -n kube-system -l k8s-app=<cni-name>

# Check CNI pod logs
kubectl logs -n kube-system <cni-pod> -c <container>

# On the node, check CNI plugin binary
ls -la /opt/cni/bin/
ls -la /etc/cni/net.d/

Check Resource Pressure

# View node resource usage
kubectl top nodes
kubectl top node <node-name>

# Check kubelet resource reservation
kubectl describe node <node-name> | grep "Allocated resources" -A 5

# On node: check disk usage
df -h
du -h /var/lib/kubelet/
du -h /var/log/

# On node: check memory
free -h

# On node: check process count
ps aux | wc -l

Drain and Cordon

# Mark node as unschedulable (no new pods)
kubectl cordon <node-name>

# Evict all pods gracefully
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Uncordon when fixed
kubectl uncordon <node-name>

Common Pitfalls

Pitfall: Node NotReady, Kubelet Crashed Node shows NotReady; new pods cannot be scheduled.

• Why: kubelet process crashed or was killed. Container runtime may still be running, but kubelet is not communicating with API server or managing containers.
• Fix: SSH to node. Check kubelet status: sudo systemctl status kubelet. Check logs: sudo journalctl -u kubelet | tail -100. Restart: sudo systemctl restart kubelet. Look for error patterns (OOMKilled, cgroup mismatch, certificate expired, etc.).

Pitfall: Node Ready, But Pods Hang in ContainerCreating Node is Ready; kubelet is communicating; but pods cannot start.

• Why: Container runtime crashed or is unresponsive, or CNI plugin failed. kubelet cannot create containers (runtime unreachable) or assign networking (CNI broken).
• Fix: Check container runtime: crictl info. If hangs, runtime is broken; restart containerd/CRI-O. Check CNI DaemonSet: kubectl get pods -n kube-system -l k8s-app=<cni>. Verify CNI pod is Running. Check CNI logs.

Pitfall: Kubelet Crash-Loops with Cgroup Errors Post-Upgrade After upgrading kubelet, service repeatedly crashes with cgroup-related errors.

• Why: Kubelet cgroup driver setting differs from container runtime. Likely: kubelet was upgraded to systemd driver, but container runtime still uses cgroupfs.
• Fix: SSH to node. Check kubelet config: cat /etc/kubernetes/kubelet/kubeadm-flags.env | grep cgroup. Check container runtime: crictl info | grep cgroupDriver. Ensure both match (both systemd or both cgroupfs). Upgrade container runtime to match, or change kubelet config and restart.

Pitfall: Node Under MemoryPressure, Pods Evicted Unexpectedly Node RAM exhausted; pods are being evicted.

• Why: Node allocated more memory to pods than physically available. kubelet evicts pods to free memory, starting with BestEffort (no resource requests), then Burstable.
• Fix: Add memory to node, or reduce pod requests/reservations. Scale out to more nodes. Investigate which pods are consuming excessive memory: kubectl top pods -A | sort -k3 -rn.

Pitfall: NetworkUnavailable Taint Blocks Pod Scheduling Brand-new node shows NetworkUnavailable: True; pods don’t schedule even after waiting.

• Why: CNI DaemonSet pod has not started or has not finished initializing. This is normal for the first 30 seconds. If persists beyond that, CNI pod is stuck (crashlooping or pending).
• Fix: If new node, wait 30 seconds and verify: kubectl describe node <node> | grep NetworkUnavailable. If still True after 30s, check CNI DaemonSet: kubectl get pods -n kube-system -l k8s-app=<cni>. Verify it’s Running, not Pending/CrashLoop. Check logs.

Pitfall: Disk Pressure Accumulation Node shows DiskPressure: True; kubelet stops scheduling pods; old logs fill the disk.

• Why: Container logs accumulate in /var/log/pods/, or kubelet state in /var/lib/kubelet/ grows unbounded. Kubelet evicts pods when disk < 15% free (default threshold).
• Fix: SSH to node. Free space: sudo du -h /var/log/ | sort -h | tail -5 (identify large log directories), manually clean old logs. Check kubelet config for --eviction-hard=nodefs.available<15%. Add disk space or configure log rotation.

Interview Questions

Q — A node shows NotReady. What are the three most likely causes and how would you diagnose each? Three most likely causes: (1) kubelet crashed—check sudo systemctl status kubelet and sudo journalctl -u kubelet. (2) Container runtime crashed—test with crictl info, which is independent of Kubernetes; if it hangs, runtime is broken. (3) Networking broken—check CNI DaemonSet pods: kubectl get pods -n kube-system -l k8s-app=<cni>. All three cause NotReady because kubelet cannot register Ready status (or API server connectivity is broken, or pods can’t start). Diagnosis requires direct node access via SSH to check systemd logs and runtime health.

Q — Pods are stuck in ContainerCreating on a Ready node. What’s happening, and how would you fix it? kubelet is running and Ready, but cannot create containers. The container runtime is either crashed or unresponsive. Fix: SSH to node, run crictl info. If this hangs, the runtime is frozen. Restart it: sudo systemctl restart containerd (or CRI-O). If crictl info works, the issue is likely the CNI plugin—check if CNI DaemonSet pod is Running: kubectl get pods -n kube-system -l k8s-app=<cni>. If not Running, troubleshoot the CNI pod (logs, resource limits, etc.).

Q — Why do nodes with cgroup driver mismatch between kubelet and container runtime fail? How would you detect and fix this? kubelet and container runtime communicate about resource management via cgroups. If they use different cgroup drivers (kubelet systemd, runtime cgroupfs), kubelet cannot properly track or limit container resources, causing crash-loops with cgroup-related errors. Detect: crictl info | grep cgroupDriver and cat /etc/kubernetes/kubelet/kubeadm-flags.env | grep cgroup-driver. Fix: Ensure both use the same driver. Upgrade the component that’s behind, or change kubelet’s config and restart.

Q — How does Kubernetes evict pods under memory pressure? Which pods are evicted first? Under MemoryPressure, kubelet evicts pods to free memory in QoS order: BestEffort pods (no resource requests/limits) are evicted first; then Burstable (requests < limits); Guaranteed pods (requests == limits) are evicted last. This ensures critical workloads (Guaranteed) survive resource pressure longest. Eviction is not forced immediately—there’s a grace period allowing pods to shut down gracefully. Monitor: kubectl describe node <node> shows evicted pods in “Evicted” status.