Autoscaling

TL;DR

• HPA scales the number of pod replicas based on metrics (CPU, memory, custom) — requires metrics-server or external metrics adapter
• VPA scales individual pod resource requests/limits based on historical usage; operates independently of HPA
• Cluster Autoscaler scales the number of nodes when pods are Pending due to insufficient capacity; respects PodDisruptionBudgets
• Never run HPA (CPU/memory) and VPA (Auto mode) on the same Deployment — they interfere with each other’s decision-making
• HPA polls metrics every 15 seconds (configurable) and respects stabilization windows to prevent “flapping”
• Pending pods after HPA scale-up indicate insufficient node capacity, not an HPA failure

Core Concepts

Horizontal Pod Autoscaler (HPA)

HPA scales the number of pod replicas up or down based on observed metrics. It polls metrics every 15 seconds (default, configurable via --horizontal-pod-autoscaler-sync-period) and evaluates whether the current replica count matches the target metric threshold.

HPA requires a metrics pipeline: metrics-server for CPU and memory metrics, or a custom/external metrics adapter (Prometheus Adapter, KEDA) for custom business metrics (queue depth, request latency, etc.).

The behavior configuration block (introduced in autoscaling/v2) prevents “flapping” — rapid scaling oscillations caused by noisy metrics — through stabilization windows and scaling policies.

Vertical Pod Autoscaler (VPA)

VPA adjusts a pod’s resource requests and limits automatically based on historical actual usage patterns. Unlike HPA, VPA does not ship with core Kubernetes and must be installed separately.

VPA requires evicting and recreating pods to apply new resource values (no live in-place resizing in most cluster versions), resulting in brief service disruption each time it updates.

Critical constraint: HPA (scaling on CPU/memory metrics) and VPA (changing requests) must not target the same Deployment simultaneously. VPA’s changes to requests shift the baseline for HPA’s utilization percentage calculations, causing the two controllers to react to each other rather than actual workload changes. If both are needed, use HPA on custom metrics while VPA manages resource sizing, or run VPA in Off or recommendation-only mode.

Cluster Autoscaler (CA)

Cluster Autoscaler scales the number of nodes by monitoring Pending pods that cannot be scheduled due to insufficient node capacity. It adds nodes to node groups (AWS ASG, GCP MIG, etc.) to accommodate new pods and removes underutilized nodes when their workloads can be safely rescheduled elsewhere.

CA respects PodDisruptionBudgets, taints, and pod anti-affinity constraints when deciding whether it’s safe to drain a node before removal. It will not violate a PDB to scale down.

Interaction Between HPA and Cluster Autoscaler

These autoscalers operate in sequence: HPA scales replicas based on metrics, then Cluster Autoscaler adds nodes if pods cannot be scheduled. Without Cluster Autoscaler, HPA-driven scaling can result in Pending pods that cannot be placed due to insufficient node capacity.

Comparison Table

Configuration Reference

Horizontal Pod Autoscaler Example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

Vertical Pod Autoscaler Example

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: batch-processor-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: batch-processor
  updatePolicy:
    updateMode: "Auto"   # Options: Auto | Recreate | Initial | Off

Cluster Autoscaler Configuration

# Typical flags on cluster-autoscaler deployment
--nodes=3:20:eks-worker-nodegroup
--scale-down-delay-after-add=10m
--scale-down-unneeded-time=10m
--balance-similar-node-groups
--skip-nodes-with-system-pods=true

Command Reference

HPA Status and Diagnostics

# View HPA status and current metrics
kubectl get hpa payments-api-hpa
kubectl describe hpa payments-api-hpa

# Check detailed HPA events and scaling history
kubectl get hpa payments-api-hpa -o jsonpath='{.status}'

# Generate load to test HPA scaling behavior
kubectl run load-generator --image=busybox --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://payments-api; done"

# Verify metrics-server is operational
kubectl top pods
kubectl top nodes

VPA Inspection and Recommendations

# View VPA recommendations without applying them
kubectl describe vpa batch-processor-vpa | grep -A10 Recommendation

# List all VPAs and their current update mode
kubectl get vpa -o custom-columns=NAME:.metadata.name,MODE:.spec.updatePolicy.updateMode

# Check VPA logs for decision rationale
kubectl logs -n kube-system -l app=vpa-recommender | tail -20

Cluster Autoscaler Diagnostics

# Check Cluster Autoscaler logs for scaling decisions
kubectl logs -n kube-system deployment/cluster-autoscaler | grep -i "scale"

# View nodes grouped by availability zone or instance type
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type

# Identify nodes marked for removal by CA
kubectl describe nodes | grep -A5 "DiskPressure\|MemoryPressure"

Common Pitfalls

Pitfall — HPA Shows TARGETS= and Never Scales HPA object is created but remains in an unknown state without scaling any workloads.

• Why: The metrics pipeline (metrics-server or custom metrics adapter) is missing or broken; HPA cannot retrieve metrics to make scaling decisions.
• Fix: Verify kubectl top pods returns real numbers. Install or repair metrics-server; check its logs for errors.

Pitfall — HPA and VPA (Auto) Interfere With Each Other Running HPA on CPU utilization and VPA in Auto mode on the same Deployment causes erratic scaling behavior.

• Why: VPA changes requests, shifting HPA’s utilization percentage baseline; the controllers react to each other’s changes rather than actual metrics.
• Fix: Never run both simultaneously on CPU/memory metrics. Use HPA with custom metrics or run VPA in Off/recommendation-only mode.

Pitfall — HPA Scaled to 20 Replicas But Pods Are Pending HPA scales the replica count, but new pods cannot be placed.

• Why: HPA controls replicas, not node capacity. Insufficient allocatable resources exist to schedule all new pods.
• Fix: Deploy Cluster Autoscaler to add nodes when pods are Pending; verify nodes have available capacity with kubectl describe node.

Pitfall — Cluster Autoscaler Doesn’t Scale Down Even With No Load Nodes remain after workloads are removed.

• Why: CA respects PodDisruptionBudgets and other constraints (system pods, node selectors, anti-affinity). A pod with a strict PDB or a system pod may prevent node removal.
• Fix: Check CA logs for eviction blockers; adjust PDB policies or move system pods if CA scale-down is desired.

Pitfall — VPA Updates Pod Specifications Unexpectedly Pods are periodically evicted and recreated with different resource specs.

• Why: VPA in Auto mode periodically evicts and recreates pods based on recommendations; this causes service interruption.
• Fix: Use VPA in Off or Initial mode for visibility-only, or expect and handle the disruption (e.g., with PDB and sufficient replicas).

Interview Questions

Q — Explain how HPA and Cluster Autoscaler interact. When does each make decisions, and what happens if Cluster Autoscaler is absent?

Q — Why is running HPA on CPU metrics and VPA in Auto mode on the same Deployment problematic? What is the solution?

Q — A pod is OOMKilled after HPA scales up replicas. Is this an HPA failure? Why or why not?

Q — How does HPA’s stabilization window prevent “flapping,” and why is it important?

Q — Design an autoscaling strategy for a web application where you need both replica scaling and resource optimization without conflicts.

Q — What does Cluster Autoscaler check before removing a node, and why is it important?

Q — Given HPA with TARGETS=<unknown>, walk through the troubleshooting steps to identify the root cause.

Q — Compare the purposes of HPA, VPA, and Cluster Autoscaler, and explain why all three might be needed in a large cluster.