High Availability & Disaster Recovery

TL;DR

• RTO (how much downtime is tolerable) and RPO (how much data loss is tolerable) are the actual requirement — every DR strategy is just an implementation detail chosen to hit specific RTO/RPO numbers.
• Multi-AZ protects against a single datacenter failure within a Region; it does nothing for a regional outage. Multi-region strategies are a separate concern.
• The four multi-region DR strategies (backup and restore, pilot light, warm standby, active-active) sit on a single cost/RTO spectrum — cheaper and slower on one end, expensive and near-instant on the other.
• Active-active is the fastest failover but introduces real distributed-systems problems (cross-region data consistency, conflict resolution) the other three largely avoid.
• An untested backup or untested DR runbook is a hypothesis, not a capability — infrastructure and IAM drift silently until a drill (or a real incident) exposes it.
• “Zero downtime, zero data loss” is not a free requirement — translating it into real RTO/RPO numbers exposes its actual cost before committing to it.

Core Concepts

RTO and RPO

RTO (Recovery Time Objective): the maximum acceptable time to restore service after a failure. RPO (Recovery Point Objective): the maximum acceptable amount of data loss, measured in time (e.g., “RPO of 15 minutes” means up to 15 minutes of data since the last replication/backup point can be lost). Every DR conversation that skips defining these numbers is designing against an unstated requirement — the technical strategy cannot be selected correctly without them.

Multi-AZ

Protects against the failure of a single Availability Zone (effectively, a datacenter or small cluster of datacenters) within a Region, by maintaining a synchronously or near-synchronously replicated standby in a different AZ. It does nothing to protect against a broad Region-level service disruption — Multi-AZ and multi-region address entirely different failure domains, and treating “we’re Multi-AZ” as an answer to “what happens if the Region goes down” is a category error.

Multi-region DR strategies

Four strategies exist on a single cost/RTO/RPO spectrum:

• Backup and restore — data is backed up (typically to S3 or cross-region snapshots) and restored into a new environment only when a failover is actually triggered. Cheapest and simplest; RTO is measured in hours (time to provision infrastructure and restore data), RPO is tied directly to backup frequency.
• Pilot light — only core infrastructure (most often the database) runs continuously in the DR Region, kept in sync via replication. The rest of the stack (compute, app tier) is provisioned and scaled up from infrastructure-as-code only during an actual failover. Cheaper than warm standby; RTO includes the time to stand up the non-core stack.
• Warm standby — a scaled-down but fully functional copy of the entire stack runs continuously in the DR Region. Failover means scaling existing infrastructure up to full capacity, not building it from scratch. Faster RTO than pilot light, higher steady-state cost.
• Active-active — both Regions serve production traffic simultaneously at full scale. Failover is near-instantaneous because there is no “stand up” step at all. This is the most expensive and operationally complex option, and it introduces distributed-systems problems the other three mostly avoid: cross-region data consistency, write conflict resolution, and the need for idempotent/commutative operations or a conflict-resolution strategy (e.g., last-writer-wins, CRDTs).

The core design skill is translating a business-stated RTO/RPO into the cheapest strategy on this spectrum that actually satisfies it — not defaulting to the most sophisticated-sounding option, and being able to show the cost/complexity curve when a stakeholder asks for “zero downtime” without having priced out what that costs.

Testing: the part that’s routinely skipped

A DR plan that has never been executed as a drill is a hypothesis, not a capability. Infrastructure drifts continuously — IAM roles get deleted or reassigned, DNS records get repointed, application dependencies change. A runbook written a year ago and never re-executed may reference resources, roles, or steps that no longer match reality, and the only way to know is to actually run it. The same applies to backups: an untested restore procedure is not a validated RPO/RTO, it is an assumption.

Backup and Restore vs Pilot Light vs Warm Standby vs Active-Active

Common Pitfalls

• Pitfall: Multi-AZ is cited as the answer when asked about surviving a regional outage. Why: Multi-AZ protects against a single-datacenter failure within a Region; it provides no protection if the Region itself has a broad disruption. Fix: Design an explicit multi-region strategy (pilot light, warm standby, or active-active) matched to the actual RTO/RPO for regional-failure scenarios.
• Pitfall: “Zero downtime, zero data loss” is accepted as a requirement without discussing cost. Why: Active-active multi-region is achievable but expensive in infrastructure spend and operational complexity; the requirement as stated has a real price tag. Fix: Translate the requirement into explicit RTO/RPO numbers and present the cost curve before committing to a design.
• Pitfall: Nightly backups are treated as a complete DR plan. Why: An untested backup is a hypothesis, not a validated recovery capability, and “we have backups” is not an RTO/RPO number. Fix: Periodically test actual restores, measure restore time against the stated RTO, and derive RPO from actual backup frequency.
• Pitfall: DR runbooks reference infrastructure or IAM roles that no longer exist. Why: Infrastructure and permissions drift over time, and a runbook that isn’t regularly executed drifts out of sync silently. Fix: Schedule DR drills on a real cadence and treat a drill that surfaces a broken runbook as a successful drill — that is exactly its purpose.
• Pitfall: Active-active is chosen by default because it has the best failover characteristics. Why: It introduces real distributed-systems complexity (cross-region consistency, conflict resolution) that the business’s actual RTO/RPO may not require. Fix: Select the cheapest strategy on the spectrum that satisfies the stated RTO/RPO, not the most capable one available.

Interview Questions

• “A stakeholder says they need ‘zero downtime.’ What’s the next step in the conversation?” — tests whether real RTO/RPO numbers are pushed for instead of silently over-engineering or silently agreeing.
• “Explain the difference between pilot light and warm standby with an actual cost/RTO trade-off, not just the definitions.” — tests whether these are understood as points on a spectrum, not a fixed vocabulary list.
• “A DR runbook hasn’t been drilled in over a year. Why does that matter, and how would it be justified to leadership as worth fixing?” — tests whether the operational case (infrastructure drift) is understood, not just the technical definitions.
• “Why doesn’t Multi-AZ satisfy a requirement to survive a full regional outage?” — tests whether the failure-domain distinction between Multi-AZ and multi-region is clearly understood.
• “Design a DR strategy for a workload with an RPO of 5 minutes and an RTO of 30 minutes.” — tests whether the candidate can map concrete numbers to the cheapest strategy that satisfies them, rather than defaulting to active-active.