Control Plane Troubleshooting

TL;DR

• Control-plane failures differ fundamentally from workload failures; kubectl itself may be partially unusable; SSH to node + crictl/logs are essential
• etcd must be healthy before API server can function; API server health immediately depends on etcd availability
• All control-plane components are static pods; YAML syntax errors cause silent kubelet refusal, no error message from API server
• HA control plane: leader election ensures only one scheduler/controller-manager actively runs; stuck election causes pod to appear “Running” but do nothing
• Test API server health directly: curl -k https://localhost:6443/healthz (bypasses kubeconfig)
• Troubleshoot via: node SSH → crictl to inspect containers → kubelet/component logs in /var/log

Core Concepts

Static Pods vs. Regular Pods

Regular pods: API server manages via Deployment/DaemonSet; restarts on node loss.

Static pods: kubelet manages directly from local manifest files (e.g., /etc/kubernetes/manifests/); survive API server/etcd outage. All control-plane components are static pods.

When API server is down, kubectl cannot query or manage static pods. Only direct node access (SSH + crictl) can inspect them.

Component Startup Dependency Chain

kubelet (node agent)
    ↓
[reads static pod manifests from /etc/kubernetes/manifests/]
    ↓
etcd (must start first—API server depends on it)
    ↓
kube-apiserver (depends on etcd)
    ↓
kube-scheduler (depends on API server)
kube-controller-manager (depends on API server)

etcd must be healthy and running before kube-apiserver starts; API server startup fails if etcd is unreachable.

HA Control Plane: Leader Election

In HA clusters (3+ control-plane nodes), multiple instances of scheduler and controller-manager run across nodes. Only ONE is the active “leader”; others are standby.

Leader election mechanism: Lease object in etcd; whichever replica acquires the lease becomes leader. If leader crashes, another replica acquires the lease within seconds.

Common issue: Stuck leader election (network partition, etcd unavailable) causes all scheduler/controller-manager replicas to appear “Running” but none execute—no scheduling happens.

Static Pod Manifest Errors

Kubelet reads static pod manifests from /etc/kubernetes/manifests/ and creates pods directly. If manifest contains YAML syntax errors:

• kubelet silently skips the manifest
• No error is reported to API server (API server doesn’t manage static pods)
• Pod never starts
• Only evidence is kubelet logs: “malformed manifest” or similar

Comparison Table

Command Reference

Diagnose Control Plane

# Check control-plane node health
kubectl get nodes -o wide

# Check API server health (from control-plane node or externally)
curl -k https://localhost:6443/healthz                  # Returns "ok" or component-specific status

# Check etcd health (requires etcd certs, typically on control-plane node)
ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  -w table endpoint health

# Get cluster component status (deprecated but sometimes useful)
kubectl get componentstatuses

Access Static Pods Directly

# SSH to control-plane node, then use crictl
ssh user@control-plane-node

# List all static pods
crictl ps | grep -E "apiserver|etcd|scheduler|controller-manager"

# Inspect specific pod
crictl inspect <container-id>

# View container logs
crictl logs <container-id>

Debug Static Pod Manifests

# SSH to control-plane node

# List manifest files
ls -la /etc/kubernetes/manifests/

# View API server manifest
cat /etc/kubernetes/manifests/kube-apiserver.yaml

# Check kubelet logs for manifest parsing errors
sudo journalctl -u kubelet | grep -i manifest

# Look for kubelet errors
sudo journalctl -u kubelet | tail -100

Inspect Component Logs

# On control-plane node, check component logs
sudo tail -f /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log
sudo tail -f /var/log/pods/kube-system_kube-scheduler-*/kube-scheduler/*.log
sudo tail -f /var/log/pods/kube-system_etcd-*/etcd/*.log

Common Pitfalls

Pitfall: API Server Unresponsive, kubectl Unusable Cluster appears down; kubectl commands hang or fail; cannot diagnose further.

• Why: If API server is down or unreachable, kubectl cannot connect; all commands fail. kubectl itself is not broken—it’s the API server that’s unavailable.
• Fix: SSH directly to a control-plane node. Use crictl ps to inspect control-plane static pods directly. Check pod logs via crictl logs or /var/log/pods/. Test API server health: curl -k https://localhost:6443/healthz. Check etcd status independently.

Pitfall: Static Pod Manifest YAML Syntax Error Control-plane component simply disappears; no clear error message.

• Why: kubelet reads static pod manifests from /etc/kubernetes/manifests/ directly. If manifest has YAML syntax error, kubelet silently skips it. No error is reported to API server (which isn’t managing the static pod anyway). Pod never starts; only evidence is kubelet logs.
• Fix: SSH to control-plane node. Check kubelet logs: journalctl -u kubelet | grep manifest. Verify manifest syntax: kubectl apply --dry-run=client -f /etc/kubernetes/manifests/<manifest>.yaml or use a YAML validator. Fix manifest, kubelet auto-detects change and recreates pod.

Pitfall: HA Scheduler Appears “Running” but Nothing Schedules Multiple scheduler replicas running; pods remain Pending; no errors in scheduler logs.

• Why: Stuck leader election in HA cluster. All scheduler instances are running, but none has acquired the leader lease in etcd. Without the lease, a scheduler is a “standby” and doesn’t actively schedule. If etcd is unavailable or network partition exists, no scheduler can acquire the lease.
• Fix: Check etcd health: etcdctl endpoint health. Verify network connectivity between control-plane nodes. Check scheduler leader lease: kubectl get lease -n kube-system -l component=scheduler. Restart problematic control-plane node or manually delete the stuck lease to trigger re-election.

Pitfall: etcd Unhealthy, API Server Still Claims “Ready” API server pod is running and ready; but API calls hang or fail.

• Why: API server startup succeeds even if etcd briefly becomes unhealthy after startup. Liveliness probe may pass (API server process is alive) but readiness can be affected by etcd availability. API server log shows connection errors to etcd.
• Fix: Verify etcd is healthy: etcdctl endpoint health. Check API server logs for etcd errors: crictl logs <apiserver-container-id>. Restart etcd if necessary (on control-plane node via static pod restart).

Pitfall: Leader Election Lease Stuck In HA cluster, one component takes the lease but crashes; other instances don’t take over.

• Why: Leader lease is stuck if the holder crashes and doesn’t cleanly release the lease. Lease has a TTL; other replicas can acquire it after TTL expires, but delay is 15-30 seconds. Immediate recovery requires manual lease deletion.
• Fix: Delete the stuck lease: kubectl delete lease <component>-leader -n kube-system. New election triggers immediately; another replica acquires lease and becomes leader.

Interview Questions

Q — Control-plane outage occurs; kubectl is partially unusable. How would you diagnose the issue when you can’t rely on kubectl commands? SSH directly to a control-plane node (you should have direct access or bastion). Use crictl ps to list control-plane static pods; check if kube-apiserver, etcd, scheduler, and controller-manager containers are running. Use crictl logs <container-id> to view component logs. Check kubelet logs via journalctl -u kubelet. Test API server directly: curl -k https://localhost:6443/healthz. For etcd issues, use etcdctl endpoint health (requires cert access). Look for error messages indicating which component failed and why.

Q — What is the startup dependency chain for control-plane components? Why must etcd start before API server? etcd is the backing data store; without it, API server has nowhere to store or retrieve cluster state. API server startup probes for etcd availability; if etcd is unreachable, API server fails to start or cannot serve requests. The chain is: kubelet (node agent) → reads static pod manifests → starts etcd → API server starts (depends on etcd) → scheduler and controller-manager start (depend on API server). Breaking etcd breaks the entire chain downstream.

Q — In an HA control plane with multiple scheduler replicas, what does leader election mean? What happens if leader election is stuck? In HA, only ONE scheduler instance is active and scheduling; others are standby. A lease object in etcd is used for leader election—whichever scheduler replica holds the lease is the leader. If the leader crashes, another replica acquires the lease (after TTL) and becomes leader. If leader election is stuck (e.g., network partition preventing lease acquisition), all scheduler replicas are standby, none actively schedules, and pods remain Pending. Fix: Manually delete the stuck lease to force immediate re-election.

Q — Static pod manifest has a YAML syntax error. Why is this error hard to diagnose? Static pods are not managed by API server; kubectl cannot inspect them. Kubelet reads manifests directly from /etc/kubernetes/manifests/. If manifest has syntax error, kubelet silently skips it—no API server error, no pod created, no clear error message. Only evidence is in kubelet logs (search for “manifest” or the component name). This is why direct node access (SSH + journalctl -u kubelet) is essential for static pod troubleshooting.