Production-Grade Architecture Scenarios

TL;DR

• Senior-level cloud architecture questions are deliberately ambiguous — the correct answer is determined by requirements, constraints, risk tolerance, cost, and operational maturity, not by memorized service lists.
• A production-grade answer always covers five paths: request, identity, data, failure, and operator (who debugs this at 3 a.m.).
• The review bar is: survives one AZ failure unattended, debuggable by a new engineer from dashboards/traces/logs/runbooks alone, provably least-privilege and encrypted, and cost-attributable in both steady state and failure mode.
• Biggest gotcha: architectures that only describe the happy path or that maximize resilience without evidence the business needs or can operate that level of resilience both fail review.

Core Concepts

Scenario Method

• Start every scenario with explicit requirements: user count and geography, latency targets, RTO/RPO, compliance regime, data residency, traffic shape (steady vs. bursty), and team ownership boundaries.
• Draw five paths before selecting services: request path (how traffic reaches compute), identity path (how principals are authenticated/authorized), data path (where state lives and how it flows), failure path (what happens when a dependency dies), and operator path (how a human observes and intervenes).
• State assumptions explicitly before naming AWS services, then name the alternatives that were considered and rejected, with the specific reason each was rejected (cost, operational complexity, compliance mismatch, latency).

FAANG-Style Review Bar

A design clears review only if it can answer yes to all of:

• Does it survive a single-AZ failure without manual intervention?
• Can an engineer who did not build it debug a live incident using only dashboards, traces, logs, and runbooks?
• Can the security team prove least privilege, encryption in transit and at rest, auditability of every privileged action, and tenant isolation?
• Can finance attribute cost for both steady-state operation and degraded/failure-mode operation (e.g., cross-AZ replication cost during failover)?

Scenario Catalog

Each of the following is a full whiteboard exercise: state requirements, describe steady-state architecture, describe failure mode and mitigation, enumerate security controls, and produce an operational runbook outline.

• Three-tier payment platform (see architecture-03)
• Custom domain auth and API edge (see architecture-04)
• Secure GenAI agent platform (see architecture-05)
• Enterprise landing zone (see architecture-07)
• Multi-region event-driven SaaS (see architecture-06)
• Hybrid zero-trust migration (see architecture-07)

Whiteboard Structure vs. Narrative Answer

Common Pitfalls

• Pitfall: Listing AWS services without justification. Why: Interviewers and reviewers cannot verify the design decisions actually map to the stated requirements. Fix: For every service named, state the specific requirement it satisfies.
• Pitfall: Designing only the happy path. Why: Production incidents happen in degraded mode, not steady state, so an undescribed failure path means the design is untested on paper. Fix: Explicitly walk through what happens when the primary dependency is unavailable.
• Pitfall: Omitting tenant isolation, identity boundaries, certificate lifecycle, and audit evidence. Why: These are the controls that actually get exercised during a security review or compliance audit. Fix: Name the specific mechanism (IAM policy, KMS key policy, cert rotation automation, CloudTrail) for each.
• Pitfall: Defaulting to maximum resilience (e.g., active-active multi-region) regardless of requirements. Why: Higher resilience tiers carry real operational and cost overhead that may not be justified by the business’s actual RTO/RPO. Fix: Size the resilience tier to the stated recovery objectives, not to what sounds impressive.

Interview Questions

• What is the user-visible impact when the most important dependency fails? — Tests whether the candidate has actually traced the failure path, not just the happy path.
• Which controls are preventive, which are detective, and which are responsive? — Tests security-control taxonomy and whether the candidate can categorize their own design.
• What data is replicated, encrypted, retained, or intentionally not stored? — Tests data governance and whether “we store everything” was a conscious decision.
• What dashboard, alarm, trace, or audit event proves the system is healthy? — Tests whether observability was designed in, not bolted on.
• What is the rollback path when a deployment, migration, or failover goes wrong? — Tests operational maturity beyond the initial rollout plan.