ETCD Backup & Restore

TL;DR

• etcd is the single source of truth for cluster state — every Deployment, Secret, RBAC rule, and Service object lives there. Loss of etcd with no backup means total cluster state loss, even if workloads keep running briefly on their own.
• etcdctl snapshot save produces a point-in-time backup file; etcdctl snapshot restore never modifies the running data directory — it always creates a new one that must be pointed at explicitly.
• All snapshot operations require mutual TLS certificates (--cacert, --cert, --key) because etcdctl talks to etcd exactly like any other cluster component.
• A restore reverts the cluster to the exact state at snapshot time — anything created, changed, or deleted afterward is permanently lost.
• Verifying a snapshot is valid (snapshot status) is not optional — a save command can exit successfully while producing a corrupted or truncated file.
• This is one of the most heavily tested CKA topics.

Core Concepts

What etcd Stores and Why Backup Matters

etcd is a distributed, consistent key-value store and the backing database for the entire Kubernetes API — every object (Pods, Deployments, Secrets, ConfigMaps, RBAC bindings, Services) is serialized and stored there. The API server is stateless with respect to cluster data; it reads and writes through to etcd. If etcd’s data is lost with no backup, the cluster’s desired-state record is gone — existing running workloads may continue operating briefly since kubelets keep executing already-scheduled pods, but no reconciliation, scheduling, or API operations can function correctly, and recovery without a backup means rebuilding every object definition from scratch.

Taking a Snapshot

etcdctl snapshot save produces a single file capturing the entire keyspace at that instant. Because etcdctl communicates with the etcd server over mutual TLS, exactly like kube-apiserver does, every snapshot operation requires three certificate flags: --cacert (trusted CA), --cert (client certificate), and --key (client private key). In a stacked-etcd kubeadm cluster these live under /etc/kubernetes/pki/etcd/.

A snapshot file existing on disk does not guarantee it is usable. etcdctl snapshot status <file> --write-out=table independently reads and validates the file’s internal consistency (hash, revision, key count) — this check should run immediately after every snapshot, not just be assumed from the save command’s exit code.

Restoring From a Snapshot

etcdctl snapshot restore never touches the currently running etcd data directory. It always creates a brand-new data directory populated from the snapshot contents. To make etcd actually use the restored data, the static pod manifest’s hostPath volume for the etcd data directory must be edited to point at the new path. Because etcd runs as a static pod (managed directly by the kubelet from a manifest file, not through the API server), saving the edited manifest causes the kubelet to automatically detect the change and recreate the pod — no manual pod deletion is required.

Full Disaster Recovery Sequence

1. Stop the API server temporarily, or accept that it will error until etcd is back — commonly done by moving static pod manifests out of /etc/kubernetes/manifests/ momentarily.
2. Restore the snapshot to a new data directory with etcdctl snapshot restore.
3. Update etcd.yaml’s volume path to point at the new directory.
4. Move manifests back (or let the kubelet recreate the static pods automatically).
5. Verify cluster health: kubectl get nodes, kubectl get pods -A, and confirm objects created after the snapshot are indeed gone — this is expected, not a bug.

Restore Consequences

A restore reverts the cluster to the exact state captured at snapshot time. Any Deployments, scaling events, Secret rotations, or deletions that happened after the snapshot was taken are completely gone, and the cluster will actively reconcile back toward that older desired state — for example, scaling a Deployment back down if it was scaled up after the snapshot. This makes a restore a recovery to a known-good point-in-time, not a seamless continuation; the data-loss window must be communicated and accounted for operationally.

etcd Snapshot vs Full Cluster Backup

Command / Configuration Reference

Take a snapshot:

ETCDCTL_API=3 etcdctl snapshot save /opt/backups/etcd-snapshot-$(date +%Y%m%d-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Verify the snapshot is valid:

ETCDCTL_API=3 etcdctl snapshot status /opt/backups/etcd-snapshot-20260707.db --write-out=table

Restore from a snapshot into a new data directory:

ETCDCTL_API=3 etcdctl snapshot restore /opt/backups/etcd-snapshot-20260707.db \
  --data-dir=/var/lib/etcd-restored

Point etcd at the restored directory:

# Edit /etc/kubernetes/manifests/etcd.yaml
# Change hostPath volume for etcd-data from /var/lib/etcd to /var/lib/etcd-restored

Check current etcd cluster health and member list:

ETCDCTL_API=3 etcdctl endpoint health --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

ETCDCTL_API=3 etcdctl member list --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

Find the exact etcd endpoint and cert paths from the running static pod manifest:

cat /etc/kubernetes/manifests/etcd.yaml | grep -E "cert-file|key-file|trusted-ca-file|listen-client-urls"

Automate periodic snapshots via cron:

0 * * * * ETCDCTL_API=3 etcdctl snapshot save /opt/backups/etcd-$(date +\%Y\%m\%d-\%H\%M).db --cacert=... --cert=... --key=...

Common Pitfalls

• Pitfall: A cron-scheduled snapshot job reports success for months while actually producing unusable files. Why: snapshot save can exit with a success code while still producing a subtly corrupted or truncated file due to disk space exhaustion, permission issues mid-write, or interrupted I/O — the exit code alone does not guarantee integrity. Fix: Run etcdctl snapshot status after every snapshot and alert on failures; periodically test-restore a snapshot into a scratch environment to confirm the entire pipeline works end-to-end.
• Pitfall: Running etcdctl snapshot save with the wrong endpoint or mismatched/expired certificates and silently discarding the error in a wrapper script. Why: TLS handshake failures are clear at the command line but get swallowed when scripts redirect or ignore stderr, masking the failure until an actual disaster reveals the backups never worked. Fix: Never suppress etcdctl output in automation; surface and alert on non-zero exit codes explicitly.
• Pitfall: Assuming snapshot restore updates the running cluster automatically. Why: snapshot restore only creates a new data directory on disk — it never touches the live etcd process or its currently configured data path. Fix: Explicitly edit the static pod manifest’s hostPath to point at the new directory, which the kubelet then picks up automatically.
• Pitfall: Expecting post-snapshot changes to survive a restore. Why: A restore is a point-in-time revert; the cluster reconciles back toward the exact state captured at snapshot time, discarding everything after. Fix: Take snapshots frequently enough to bound acceptable data loss, and communicate the snapshot’s timestamp as the recovery point objective during an incident.

Interview Questions

• You restore an etcd snapshot taken two hours ago after a corruption incident. What state does the cluster come back in, and what are the operational consequences? — Tests understanding of point-in-time recovery and reconciliation behavior.
• Why is it critical to verify a snapshot immediately after taking it, rather than trusting snapshot save’s exit code alone? — Tests awareness that backup validity is not implied by command success.
• Why does etcdctl snapshot save require --cacert, --cert, and --key? — Tests understanding of etcd’s mutual-TLS security model.
• Does etcdctl snapshot restore modify the running etcd data directory? If not, what additional step is required? — Tests hands-on familiarity with the actual restore mechanics and static pod behavior.
• How would you design an automated backup strategy for etcd that guarantees backups are actually restorable, not just present? — Tests operational maturity beyond rote command knowledge.