33. Resilience Engineering
Resilience Engineering
TL;DR
- Distributed systems fail continuously — networks drop packets, dependencies overload, instances die. Resilience design assumes this from the start rather than reacting to the first outage.
- Naive retries without backoff and jitter amplify outages instead of absorbing them (“retry storms”).
- Idempotency keys are the difference between a safe retry and a duplicate charge — required for any operation with a real-world side effect.
- Dead-letter queues (DLQs) are the only thing standing between “message failed” and “data silently vanished.”
- Bulkheads (fault isolation) stop one bad dependency from starving calls to unrelated healthy ones.
- A resilience design that has never been tested against injected failure (chaos engineering) is a hypothesis, not a validated capability.
Core Concepts
Retries and retry storms
Transient failures — a network blip, a momentary dependency overload — are normal in distributed systems, so retries are necessary. The failure mode is how retries are implemented: a tight loop with no delay means every failed request retries immediately, which amplifies load on an already-struggling downstream service. This is a retry storm: it can turn a brief blip into an extended outage because the retry traffic itself becomes the load that prevents recovery.
Exponential backoff and jitter
Exponential backoff spaces out retry attempts with increasing delay (e.g., 1s, 2s, 4s, 8s), reducing amplification. Backoff alone has a subtler failure mode: without jitter (randomizing the delay), clients that failed at the same moment retry at the same calculated intervals in near-lockstep, producing synchronized retry waves that hit the recovering service simultaneously — a thundering-herd effect. Jitter (adding randomness to each computed delay) spreads that timing out so retries arrive staggered instead of in waves. “Exponential backoff” without jitter is a common half-implementation: it looks correct in code review but still causes synchronized-retry problems at scale.
Idempotency
Idempotency matters most for any operation with a real-world side effect — a payment, an inventory decrement, a notification send. An idempotency key is a client-generated unique identifier for a specific logical operation, checked server-side before processing. Without one, a retried request that actually succeeded the first time (e.g., the server processed it but the response was lost before confirming back to the client) gets processed again — a duplicate charge, a double-decremented inventory count. This gap is easy to miss because it produces no failures in normal testing; it only manifests when a retry happens to coincide with a request that succeeded server-side but failed to confirm.
Dead-letter queues (DLQs)
A DLQ is the safety net for messages that exhaust their retry budget. Without one, a message that fails processing repeatedly is simply dropped — invisible unless someone notices missing data downstream. With a DLQ, that failure is preserved for investigation instead of silently vanishing. A DLQ with no monitoring attached provides effectively the same protection as no DLQ at all — the value comes from someone actually consuming and acting on what lands there.
Fault isolation (bulkheads)
Bulkheads assign separate resource pools (connection pools, thread pools, capacity) per dependency, so a failing or slow dependency can’t exhaust shared resources and take down calls to unrelated, healthy dependencies. Without isolation, one dependency’s degradation cascades into unrelated request paths purely because they shared infrastructure (a connection pool, a thread pool) with the failing one.
Chaos engineering
Chaos engineering is the deliberate injection of failure — killing an instance, introducing network latency, throttling a dependency — to validate that resilience design actually works rather than assuming it works because it was designed carefully. Mature organizations run these experiments in production (not just staging) under guardrails: limited blast radius and automatic abort conditions. The reason production testing matters: staging rarely replicates production’s real traffic patterns, scale, and dependency behavior closely enough to surface the same failure modes. A resilience design never tested against real injected failure is, like an untested DR plan, an unvalidated hypothesis.
Retry vs Circuit Breaker vs Bulkhead
| Aspect | Retry (with backoff+jitter) | Circuit Breaker | Bulkhead |
|---|---|---|---|
| Problem solved | Transient, short-lived failures | Sustained failures — stop calling a dependency that’s clearly down | Cross-contamination between dependencies |
| Mechanism | Re-attempt with increasing, randomized delay | Trip open after failure threshold, short-circuit calls, periodically probe (half-open) to test recovery | Separate resource pools (threads, connections) per dependency |
| Risk if misused | Retry storms if no backoff/jitter | Trips too aggressively on noisy failures, or never trips and lets calls pile up | Under-provisioning a pool starves a legitimately high-traffic dependency |
| Failure it does NOT address | Sustained outages (retries alone just delay the same overload) | Doesn’t prevent shared-resource exhaustion across dependencies | Doesn’t reduce the failure rate of the dependency itself |
Common Pitfalls
- Pitfall: Tight-loop retries with no backoff. Why: Every failed request retries immediately, amplifying load on an already-struggling dependency. Fix: Implement exponential backoff with jitter on every retry path.
- Pitfall: Backoff implemented without jitter. Why: Clients that fail simultaneously retry in lockstep, producing synchronized waves against the recovering service. Fix: Add randomization to each computed delay so retries spread out over time.
- Pitfall: Payment or inventory operations with no idempotency key. Why: The server can’t distinguish a retry of a successful request from a genuinely new request. Fix: Require a client-generated idempotency key for any operation with a real-world side effect, and check it server-side before processing.
- Pitfall: No DLQ, or a DLQ nobody monitors. Why: Failed messages are dropped silently, or preserved but never reviewed. Fix: Attach a DLQ to every consumer with retry logic, and wire up active monitoring/alerting on it.
- Pitfall: Resilience design that looks correct on paper but is untested. Why: Code review can’t catch emergent failure modes under real load and real network conditions. Fix: Run chaos/fault-injection experiments, ideally in production, with guardrails.
Interview Questions
- “A downstream outage got worse after your service started retrying failed calls more aggressively. What happened, and how do you fix the retry logic?” — tests whether retry storms and backoff-with-jitter are genuinely understood.
- “A payment API processed the same charge twice after a client retried a timed-out request. Walk me through the fix.” — tests whether idempotency keys are the immediate, correct answer.
- “How would you convince a risk-averse organization to start running chaos engineering experiments in production?” — tests the case for guardrailed production testing over staging-only validation.
- “When would you choose a circuit breaker over relying on retries alone?” — tests understanding of sustained vs. transient failure and when retrying just adds load to a dead dependency.