SREOperations & Culture·EXPERT·6 min read

Chaos Engineering & Reliability

TL;DR

Chaos Engineering is structured hypothesis-driven testing, not random failure injection. Hypothesis → Experiment → Measure → Fix.
Control blast radius: start in staging, move to production with 1% traffic, always maintain an instant abort mechanism.
Game Days are scheduled team exercises simulating major failures (region down, database gone) to validate runbooks and alerting under pressure.
Three key tools: Chaos Mesh/Litmus for Kubernetes, AWS FIS for cloud resources, Gremlin for enterprise SaaS chaos.
Production chaos reveals resilience gaps that staging never will—scale, traffic patterns, and configuration drift differ drastically.
Establish baseline observability (metrics, dashboards, alerts) before injecting faults; without it, experiment results are uninterpretable.

Core Concepts

Chaos Engineering Definition

Discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions. Not random breaking—structured, hypothesis-driven scientific method.

Hypothesis-Driven Experimentation

Structured approach: state a hypothesis (e.g., “database failover completes in <30s”), run the experiment (kill primary node), measure outcomes (did it failover? within SLI?), identify gaps, fix them.

Blast Radius Control

The scope of impact on real users. Uncontrolled blast radius causes user-facing outages. Mitigation: test in staging first, use traffic routing (service mesh) to isolate in production, maintain instant abort buttons.

Game Day

Scheduled team exercise simulating catastrophic failures (region outage, database loss, network partition) to validate failover mechanisms, train operators on runbooks and dashboards, identify observability gaps.

Observability Baseline

Steady-state metrics and alert thresholds established before chaos injection. Without baseline, it’s impossible to measure the impact of experiments or verify recovery.

Experiment Progression: Blast Radius

Phase	Environment	Traffic Impact	Control Mechanism	When to Escalate
Phase 1: Research	Staging	0% (internal)	Manual rollback	Obvious bugs found
Phase 2: Validate	Staging	0% (internal)	Automated rollback	Experiment succeeds
Phase 3: Small Scale	Production	1% (canary)	Service mesh (header routing)	No user impact
Phase 4: Scale Up	Production	5–10% (region/subset)	Service mesh + automated abort	Controlled success
Phase 5: Full Scale	Production	100%	Instant kill switch	Deep confidence

Comparison: Chaos Tools & Use Cases

Tool	Platform	Best For	Scope	Cost
Chaos Mesh	Kubernetes	Pod failures, network chaos, CPU/memory injection	Cluster-level	Open source
Litmus Chaos	Kubernetes	GitOps-driven chaos, CRD-based experiments	Cluster-level	Open source
AWS FIS	AWS	EC2 termination, AZ outages, API throttling	Infrastructure	Pay-per-run
Gremlin	Multi-cloud	Enterprise chaos, scheduling, compliance	Broad (VMs, containers, networks)	Enterprise SaaS

Common Chaos Experiments & Expected Outcomes

Experiment	Hypothesis	Expected Behavior	Common Failure Mode
Kill primary DB node	Failover to replica in <30s, no data loss	App continues serving, connection pool reconnects	Replica takes >2min to catch up; stale reads occur
Crash Redis cache	Degrade to direct DB queries	Latency spike, no error; circuit breaker prevents cascade	Database CPU maxes (thundering herd); cascade failure
Inject 500ms latency	Timeout detection works; fallback activates	Request times out, circuit opens, fallback serves cached data	No timeout logic; queue backs up; memory exhaustion
Terminate AZ	Cross-AZ failover within SLA	Traffic reroutes, auto-scaling activates	DNS cache TTL; RLS sticky sessions; stateful data loss
Drop 10% packets	Retries succeed; application stays healthy	HTTP 5xx retried; TCP retransmit works	No retry logic; request fails; user-facing error

Configuration Reference: Chaos Mesh Example

Kill a database pod to test failover:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: db-failover-test
  namespace: production
spec:
  action: kill  # kill, pod-kill, pod-failure
  mode: one     # one pod (blast radius control)
  selector:
    namespaces:
      - production
    labelSelectors:
      app: postgres-primary
  duration: 5m
  scheduler:
    cron: "0 2 * * 0"  # Run weekly at 2am Sunday

Inject network latency to test timeout behavior:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: cache-latency-test
spec:
  action: delay
  mode: percentage
  selector:
    namespaces:
      - production
    labelSelectors:
      app: redis-cache
  delay:
    latency: 500ms
    jitter: 100ms
  duration: 10m

Common Pitfalls

Pitfall	Why It Happens	Fix
Chaos without baseline observability	No metrics/alerts in place before injecting faults. Can’t measure impact or verify recovery.	Establish baseline: dashboard showing steady-state latency, error rate, resource usage. Create alerting rules. Then start chaos experiments.
Uncontrolled blast radius	Kill the entire database instead of one pod. Impacts production users directly.	Use service mesh (Istio) to route only test traffic to fault-injected targets. Start with 1% traffic in staging. Manual approval before production.
No abort button	Experiment reveals cascading failure; no way to stop quickly. System collapses.	Every chaos experiment must have a documented instant kill-switch (API call, config flag, traffic drain). Test the abort button first.
Staging-only chaos	Staging doesn’t replicate production scale, traffic patterns, or drift. Experiments pass staging but fail production.	Run chaos in production after validation in staging. Use feature flags + traffic routing to limit blast radius. Accept the risk.
One-time experiments	Run chaos once, document results, never run again. No regression testing.	Schedule chaos experiments as recurring (e.g., weekly game days). Version control experiment configs. Track results over time for reliability trends.
Testing happy paths only	Focus on “everything recovers perfectly”; never test cascading failures or combination outages.	Test failure combinations: “What if Redis down AND primary DB slow?” Test degradation modes: “What if 50% of data center slow?”

Interview Questions

Q — You want to run a chaos experiment: “What happens if Redis crashes?” Expected behavior vs. what might go wrong?

— Hypothesis: Application catches connection timeout, degrades to direct database queries (cache miss fallback). Expected: Latency spikes, but no errors; circuit breaker prevents cascade. What goes wrong: 100% traffic instantly hits database. Database CPU hits 100%, crashes (thundering herd). Proves need for rate limiting + circuit breaker before database, not just fallback logic. Run in staging first (kill Redis on 1% canary traffic), measure, then production with instant abort button ready.

Q — Why run chaos in production if staging is safer?

— Staging rarely replicates production’s scale, traffic patterns, and configuration drift. Auto-scaling behavior, true cross-AZ latency, and DNS cache effects only visible in production. Chaos in staging validates happy-path recovery; chaos in production proves actual resilience. Mitigate risk: small blast radius (1% traffic), instant abort, weekday off-peak hours, full observability active.

Q — You’ve never run a game day. Design one to test multi-region failover.

— Scenario: “Primary region (us-east-1) is gone; Route 53 must failover to us-west-2 within 5 min, no data loss.” Hypothesis: Automated failover works; dashboards alert immediately; RTO < 5min. Experiment: Kill 95% of us-east-1 traffic (not 100%, to avoid real user impact). Measure: Did Route 53 detect failure? Did traffic reroute? Did RDS failover? Did apps reconnect? Results: Document what worked, what failed, what alerts fired. Action items: Fix gaps. Run monthly. Track MTTR improvement.

🔒 Locked

Related in Operations & Culture

Incident Management & Postmortems

ADVANCED