StatefulSets

TL;DR

• StatefulSets guarantee stable identity: predictable pod names (db-0, db-1, db-2), stable DNS via headless Service, persistent storage per pod
• Require headless Service (clusterIP: None) for DNS names like db-0.mysql.default.svc.cluster.local
• Pods scale up in order (0 → 1 → 2); if pod-1 fails, pod-2 doesn’t start (ordered startup)
• volumeClaimTemplates create PVCs per pod; PVCs persist after StatefulSet deletion and must be cleaned up manually
• rollingUpdate.partition enables canary updates: partition=2 updates only pods with ordinal ≥ 2
• Scale-down deletes pods in reverse order (highest ordinal first); data remains in PVCs

Core Concepts

Why Not Deployments for Databases?

Deployment treats replicas as interchangeable clones. Pods get random names (db-x4j2k), random DNS, and no guaranteed storage per replica. This works for stateless services, but databases (PostgreSQL cluster, MongoDB replica set) need stable identity.

A PostgreSQL replica must always be pg-replica-0, always connect to its own PVC data-pg-replica-0, and reliably rejoin the cluster. StatefulSets guarantee this.

Stable Pod Identity

StatefulSets use sequential ordinals: <name>-0, <name>-1, <name>-2, etc. This naming is stable—if db-1 restarts, the new pod is also named db-1. Enables applications to learn member identity within the cluster.

Headless Service Requirement

StatefulSet must reference a headless Service (clusterIP: None) via spec.serviceName. This Service creates DNS records for each pod:

• db-0.mysql.default.svc.cluster.local → pod db-0’s IP
• db-1.mysql.default.svc.cluster.local → pod db-1’s IP

Without headless Service, DNS names don’t resolve; cluster discovery fails.

Ordered Pod Startup

Creation: pods 0 → 1 → 2 sequentially. Pod i+1 only starts after pod i is Ready.

Deletion: pods in reverse order (2 → 1 → 0) to avoid data corruption during cluster membership changes.

If pod-1 is CrashLoopBackOff, pod-2 never starts (waiting for pod-1 to become Ready). This prevents partial cluster members from corrupting data.

volumeClaimTemplates

StatefulSet automatically creates PVCs for each pod. Example: 3-replica StatefulSet creates:

• data-db-0 bound to pod db-0
• data-db-1 bound to pod db-1
• data-db-2 bound to pod db-2

When StatefulSet is deleted, PVCs persist (by design, to prevent accidental data loss). Manual cleanup required.

Partition-Based Canary Rollout

rollingUpdate.partition enables canary-style updates. Example: partition=2 updates only pods with ordinal ≥ 2:

• db-0, db-1 keep old template
• db-2, db-3 get new template
• Allows testing new version on subset before rolling out to all pods

Comparison Table

Command Reference

StatefulSet Manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql  # Headless service
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 50Gi

Headless Service

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  clusterIP: None  # Headless
  selector:
    app: mysql
  ports:
  - port: 3306
    targetPort: 3306

Essential Commands

# List StatefulSet pods (shows ordinal names)
kubectl get pods -l app=mysql

# Check persistent volumes created
kubectl get pvc

# Scale StatefulSet
kubectl scale statefulset mysql --replicas=5

# Rolling update with partition (canary)
kubectl patch statefulset mysql -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'

# Delete StatefulSet but keep PVCs
kubectl delete statefulset mysql --cascade=orphan

# Delete PVCs (after deleting StatefulSet)
kubectl delete pvc -l app=mysql

Common Pitfalls

Pitfall: StatefulSet Pods Don’t Get DNS Names Pods are created but nslookup pod-name.service fails.

• Why: StatefulSet has no spec.serviceName or referenced Service is not headless (has clusterIP instead of None).
• Fix: Create headless Service, set spec.serviceName in StatefulSet to match Service name. Verify Service has clusterIP: None.

Pitfall: Ordered Startup Blocks Higher Ordinals StatefulSet with 3 replicas; pod-1 crashes continuously. Pod-2 never starts.

• Why: StatefulSet enforces ordered startup; pod i+1 only starts after pod i is Ready. If pod-1 is CrashLoopBackOff, pod-2 waits indefinitely.
• Fix: Investigate pod-1 logs: kubectl logs pod-1 -n namespace. Fix the issue (resources, configuration, etc.), or delete the problematic pod to force restart and allow subsequent pods to proceed.

Pitfall: PVCs Not Deleted When StatefulSet Deleted StatefulSet deleted; PVCs remain (orphaned storage).

• Why: volumeClaimTemplates PVCs are intentionally persistent (prevent accidental data loss). Deleting StatefulSet doesn’t cascade-delete PVCs by default.
• Fix: Manually delete PVCs: kubectl delete pvc -l app=<label>. Or use kubectl delete statefulset --cascade=orphan to avoid deleting anything, then manually clean up. Document cleanup procedure.

Pitfall: Canary Partition Misunderstood Setting partition=2; expectation is to update 2 pods, but 1 gets updated.

• Why: partition means “update pods with ordinal >= partition”. partition=2 updates pods 2, 3, 4, … (not 0, 1). Partition is inclusive lower bound, not count.
• Fix: To update last 2 replicas in a 5-pod StatefulSet, use partition=3 (pods 3 and 4).

Interview Questions

Q — Why does a StatefulSet require a headless Service, and what does it enable? StatefulSet uses the referenced headless Service to create stable DNS names for each pod. A headless Service (clusterIP: None) doesn’t load-balance; instead, DNS returns all backend pod IPs directly, with SRV records per pod. This enables applications to discover and address individual replicas by name (e.g., db-0.mysql, db-1.mysql). Clustered systems (databases, message brokers) need this to establish stable member identities for replication and quorum.

Q — Explain ordered pod creation in StatefulSets. What happens if pod-1 is CrashLoopBackOff during creation? StatefulSet creates pods sequentially: 0 becomes Ready, then 1, then 2. Pod i+1 only starts after pod i reaches Ready status. If pod-1 crashes in a loop, pod-2 never starts—StatefulSet waits for pod-1 to stabilize. This prevents partial cluster initialization, which could corrupt data. Fix: diagnose and resolve pod-1’s issue, or manually delete it to force restart and allow subsequent pods to start.

Q — What are volumeClaimTemplates and why are the resulting PVCs not deleted when the StatefulSet is deleted? volumeClaimTemplates automatically create one PVC per pod. For a 3-replica StatefulSet, three PVCs are created and bound to pods 0, 1, 2 respectively. When the StatefulSet is deleted, PVCs intentionally persist (not cascade-deleted) to prevent accidental data loss. The data in PVCs remains; only the Kubernetes objects (StatefulSet, pods) are removed. Manual cleanup: kubectl delete pvc -l app=<label> or preserve PVCs for later recovery/reuse.

Q — What does rollingUpdate.partition enable, and how would you use it for a canary update? Partition specifies the lowest ordinal index to be updated. partition=2 means pods 0 and 1 keep the old template, while pods 2, 3, etc. get the new template. This enables canary updates: deploy new version to a subset (e.g., partition=2 in a 5-pod set updates pods 2-4), test it, then remove partition (or set to 0) to roll out to all pods. Allows safe validation of breaking changes before full rollout.