Highly Available Kubernetes: Multi-Master Architecture

TL;DR

• HA control plane: 3+ API server nodes, 3+ or 5-node etcd cluster (odd count for quorum), load balancer fronting API servers.
• Stacked etcd: etcd co-located on each control-plane node (simple, couples failures).
• External etcd: etcd on separate dedicated nodes (resilient, independent failure domains, more nodes).
• Quorum math: 3-node etcd tolerates 1 failure; 5-node tolerates 2 failures. 4-node still only tolerates 1 (add 5th if scaling).
• Load balancer is critical: single stable endpoint for kubectl/kubelet/controllers; single point of failure too.
• Dead etcd member removal: must explicitly etcdctl member remove before adding replacement (Raft counts it toward quorum).

Core Concepts

Stacked etcd (simple, coupled failure) vs External etcd (resilient, separate nodes).

1. Two HA Topologies

Stacked etcd (etcd co-located with each control-plane node):

┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│ Control Plane 1  │  │ Control Plane 2  │  │ Control Plane 3  │
│  apiserver       │  │  apiserver       │  │  apiserver       │
│  scheduler       │  │  scheduler       │  │  scheduler       │
│  controller-mgr  │  │  controller-mgr  │  │  controller-mgr  │
│  etcd            │◀▶│  etcd            │◀▶│  etcd            │
└─────────────────┘  └─────────────────┘  └─────────────────┘

Simpler to set up (this is what kubeadm init --control-plane-endpoint --upload-certs gives you by default), but couples etcd and control-plane failure domains — losing a node loses both simultaneously.

External etcd (etcd runs on entirely separate dedicated nodes):

┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ CP Node 1    │  │ CP Node 2    │  │ CP Node 3    │
│ apiserver    │  │ apiserver    │  │ apiserver    │
│ scheduler    │  │ scheduler    │  │ scheduler    │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       └────────────────┬┴────────────────┘
                ┌────────┴────────┐
        ┌───────┤  etcd cluster   ├───────┐
        │       │  (3+ nodes)     │       │
     ┌──┴──┐    └─────────────────┘    ┌──┴──┐
     │etcd1│                           │etcd3│
     └─────┘                           └─────┘

More resilient (etcd and control-plane components can fail independently) and allows scaling/tuning etcd’s typically stricter disk I/O requirements separately from control-plane compute needs, at the cost of more nodes to manage.

2. The Load Balancer Requirement

Every HA setup requires a load balancer (or DNS round-robin, though a real LB is strongly preferred) in front of the API servers, since kubectl, kubelet, and every other component needs one stable endpoint regardless of which specific control-plane node is currently healthy.

# Example: a simple HAProxy config balancing across 3 API servers
frontend k8s-api
    bind *:6443
    default_backend k8s-api-backend

backend k8s-api-backend
    balance roundrobin
    server cp1 10.0.0.10:6443 check
    server cp2 10.0.0.11:6443 check
    server cp3 10.0.0.12:6443 check

This endpoint is exactly what gets passed to --control-plane-endpoint during kubeadm init and baked into every certificate’s SAN list.

3. etcd Quorum: Why Odd Numbers Matter

etcd uses the Raft consensus algorithm, requiring a strict majority (quorum) of members to agree before any write succeeds. A 3-node etcd cluster tolerates losing 1 node (2/3 remain, still majority); a 5-node cluster tolerates losing 2. Critically, a 4-node cluster tolerates losing only 1 as well (needs 3/4) — adding an even-numbered node provides no additional fault tolerance while adding real cost and complexity, which is why production etcd clusters are always sized at odd numbers (3, 5, 7).

4. Hands-on Command Cheat Sheet

Check etcd cluster member health and quorum status:

ETCDCTL_API=3 etcdctl member list --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key -w table

Verify which control-plane node is currently the active scheduler/controller-manager leader:

kubectl get lease -n kube-system kube-scheduler -o yaml
kubectl get lease -n kube-system kube-controller-manager -o yaml

Test the load balancer’s failover by stopping one API server and confirming kubectl still works:

systemctl stop kubelet   # on one control-plane node, temporarily
kubectl get nodes         # should still succeed via the LB routing to a healthy node

5. Architectural Interview Q&A

Q: Why does adding a 4th node to a 3-node etcd cluster provide zero additional fault tolerance? A: Raft quorum requires a strict majority: 3 nodes need 2 to agree (tolerates 1 failure); 4 nodes need 3 to agree (still only tolerates 1 failure, since losing 2 out of 4 leaves exactly 2 remaining — not a majority). You only gain additional fault tolerance by moving to the next odd number (5 nodes, tolerating 2 failures) — even-numbered clusters simply add cost, more network chatter for consensus, and a higher chance of split-vote scenarios in a network partition, with no corresponding benefit.

Q: In a stacked-etcd HA setup, one control-plane node fails entirely (hardware failure). What is the operational risk if you don’t act relatively quickly, even though the cluster keeps functioning on the remaining two nodes? A: With stacked etcd, losing one node means losing one etcd member simultaneously — a 3-node etcd cluster now runs on exactly 2 members, meaning quorum requires both remaining members to stay healthy (2 out of the original 3 majority threshold). A second failure of either remaining node at this point takes down etcd’s quorum entirely, causing a full cluster outage where no writes can succeed at all. The practical urgency is restoring or replacing the failed node (or removing it cleanly from etcd’s member list and adding a fresh one) before a second, unrelated failure compounds into total quorum loss.

6. Common Pitfalls

[!WARNING] Forgetting to Remove a Dead etcd Member Before Adding a Replacement Simply provisioning a new control-plane node to replace a dead one, without first explicitly running etcdctl member remove for the old (dead) member, can leave etcd’s cluster membership list referencing a node that no longer exists — this skews the actual quorum math (Raft still counts the dead member toward the total cluster size when calculating majority) and can prevent the cluster from reaching quorum even with enough genuinely healthy nodes. Always cleanly remove a failed etcd member from the cluster membership before or immediately after adding its replacement.