Kubernetes Upgrades

TL;DR

• Kubernetes enforces a strict version-skew policy: kube-apiserver is the ceiling, and every other component must be at or below it, within defined offsets.
• Minor versions cannot be skipped during an upgrade — 1.27 → 1.29 directly is unsupported; the path is 1.27 → 1.28 → 1.29.
• Upgrade order is always control plane first, then worker nodes, one minor version at a time.
• Every worker node upgrade requires cordon → drain → upgrade → uncordon; skipping drain risks workload disruption instead of a clean, controlled eviction.
• kubectl drain respects PodDisruptionBudgets and will block indefinitely rather than violate one — the most common cause of a “stuck” drain.
• Only the first control-plane node runs kubeadm upgrade apply; every additional control-plane node runs kubeadm upgrade node.

Core Concepts

Version Skew Policy

Kubernetes components do not all need to run the same version, but the allowed skew is tightly bounded:

• kube-apiserver is the version ceiling — no other component may be newer than it.
• kube-controller-manager, kube-scheduler: may be up to 1 minor version older than kube-apiserver.
• kubelet, kube-proxy: may be up to 2 minor versions older than kube-apiserver.
• kubectl: may be one minor version newer or older than kube-apiserver.

This policy exists because the API server is the single source of truth for API schema, defaulting, and conversion logic; every other component is a client of it in some sense, so nothing is allowed to assume API behavior newer than what the server actually provides.

Why Minor Versions Cannot Be Skipped

Each Kubernetes minor version can introduce API deprecations, behavior changes, and internal data migrations — particularly around etcd storage schema versions and API defaulting/conversion logic. These migrations are only tested and guaranteed correct when upgrading one minor version at a time. Skipping a version means the control plane encounters stored objects or internal state with no defined migration path, because the intermediate version’s migration logic never executed. This is a deliberate safety constraint enforced by kubeadm, not an arbitrary tooling limitation.

Upgrade Order

Control plane components first (kube-apiserver → kube-controller-manager/kube-scheduler → etcd, if stacked), then worker nodes — one minor version at a time across the whole cluster. Upgrading a worker’s kubelet ahead of the control plane risks pushing it outside the supported skew window, since kubelet may only ever be older than (never newer than) kube-apiserver.

kubeadm Upgrade Mechanics

kubeadm upgrade plan performs pre-flight checks and reports the available upgrade path and any blocking issues before anything changes. Only the first control-plane node runs kubeadm upgrade apply <version>, because this command additionally applies cluster-wide configuration changes (kubeadm ClusterConfiguration, CoreDNS, kube-proxy manifests). Every subsequent control-plane node runs kubeadm upgrade node, which upgrades the local node’s static pod manifests and kubelet configuration without re-applying cluster-wide config a second time.

Cordon vs Drain in the Upgrade Flow

• kubectl cordon <node> marks the node unschedulable (SchedulingDisabled) without touching any pods currently running on it. New pods will not be scheduled there.
• kubectl drain <node> additionally evicts existing pods gracefully — respecting PodDisruptionBudgets — and lets the scheduler place them on other nodes.
• --ignore-daemonsets is required because DaemonSet-managed pods are meant to run on every node and cannot be meaningfully “evicted”; without the flag, drain refuses to proceed if any DaemonSet pods are present.
• --delete-emptydir-data is required if any pod on the node uses an emptyDir volume, since that data does not survive rescheduling and drain otherwise refuses to destroy it silently.

Node Upgrade Sequence

1. Cordon and drain the node from a control-plane node.
2. On the worker node itself: upgrade the kubeadm package, run kubeadm upgrade node, then upgrade kubelet/kubectl packages and restart the kubelet service.
3. Uncordon the node to return it to service.

kubeadm upgrade apply vs kubeadm upgrade node

cordon vs drain

Command / Configuration Reference

Check available upgrade path and pre-flight validation:

kubeadm upgrade plan

Upgrade the first control-plane node:

apt-get update && apt-get install -y kubeadm=1.29.0-1.1
kubeadm upgrade plan
kubeadm upgrade apply v1.29.0

apt-get install -y kubelet=1.29.0-1.1 kubectl=1.29.0-1.1
systemctl restart kubelet

Upgrade an additional control-plane node:

apt-get install -y kubeadm=1.29.0-1.1
kubeadm upgrade node
apt-get install -y kubelet=1.29.0-1.1 kubectl=1.29.0-1.1
systemctl restart kubelet

Safely upgrade a worker node:

# From a control-plane node
kubectl cordon worker-1
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# On the worker node
apt-get install -y kubeadm=1.29.0-1.1
kubeadm upgrade node
apt-get install -y kubelet=1.29.0-1.1 kubectl=1.29.0-1.1
systemctl restart kubelet

# From a control-plane node
kubectl uncordon worker-1

Check current versions across every component:

kubectl version
kubectl get nodes -o wide   # shows kubelet version per node

Watch pods being evicted and rescheduled live during a drain:

kubectl get pods -o wide --watch

Verify a node returned to Ready and schedulable after upgrade:

kubectl get nodes

Common Pitfalls

• Pitfall: kubectl drain hangs indefinitely on a node. Why: A PodDisruptionBudget on a workload running on that node would be violated by evicting its pod(s) — e.g. minAvailable: 1 with only one replica currently running — and drain refuses to force an eviction that violates a PDB. Fix: Scale up the affected Deployment temporarily so eviction no longer violates the PDB, or accept the availability risk explicitly via --disable-eviction (bypasses PDB enforcement; generally discouraged).
• Pitfall: Upgrading a worker node’s kubelet before the control plane is upgraded. Why: kubelet may only be older than kube-apiserver, never newer; upgrading it first can push the skew outside the supported window. Fix: Always complete and verify the control-plane upgrade (kubectl version showing the new server version) before touching any worker node.
• Pitfall: Attempting to jump multiple minor versions in one upgrade (e.g. 1.27 → 1.29). Why: Intermediate-version migration logic for API conversion and storage schema never runs, leaving no defined migration path for existing stored objects. Fix: Upgrade one minor version at a time, verifying cluster health between each step.
• Pitfall: Running kubeadm upgrade apply on a second control-plane node. Why: apply re-applies cluster-wide configuration that should only be applied once; running it again is unnecessary and can conflict with state already applied by the first node. Fix: Use kubeadm upgrade node on every control-plane node after the first.
• Pitfall: Draining a node without --ignore-daemonsets when DaemonSet pods are present. Why: DaemonSet pods are designed to run on every node and cannot be evicted in the normal sense; drain refuses to proceed. Fix: Always include --ignore-daemonsets when the node runs any DaemonSet-managed pods (which is nearly universal in real clusters).

Interview Questions

• You run kubectl drain worker-1 and it never completes. What’s the most likely cause, and how do you resolve it? — Tests understanding of PodDisruptionBudget interaction with eviction.
• Why does Kubernetes disallow skipping minor versions during an upgrade? — Tests understanding of internal migration and API conversion mechanics, not just the rule itself.
• What is the difference between kubeadm upgrade apply and kubeadm upgrade node, and when is each used? — Tests operational knowledge of multi-control-plane upgrade sequencing.
• Why must the control plane always be upgraded before worker nodes? — Tests understanding of the version-skew policy and its directionality.
• What flags are required to drain a node with DaemonSet pods and emptyDir volumes, and why does the drain command require them explicitly rather than assuming consent? — Tests command-level fluency and reasoning about data-safety defaults.