Serverless Architecture

TL;DR

• Serverless removes server management, not architectural discipline — event-driven systems make sloppy coupling and failure handling more visible, faster, because failures propagate across chains of managed services instead of staying inside one process.
• The most common anti-pattern is hand-coded orchestration (sequencing, retries, branching) buried inside Lambda function bodies instead of expressed in Step Functions, where it becomes reviewable and independently testable.
• Direct Lambda-to-Lambda invocation creates tight coupling and synchronous failure propagation; EventBridge/SNS/SQS decouples producers from consumers.
• HTTP API (API Gateway v2) is usually meaningfully cheaper than REST API (v1) for straightforward Lambda-proxy patterns that don’t need usage plans or request validation.
• Lambda Destinations catch failures that occur before a function’s own error handling ever runs — throttling, permission errors on async invocations.
• At-least-once delivery is the default assumption across these services: idempotency and dead-letter handling are not optional extras, they are required correctness properties.

Core Concepts

Lambda as the compute primitive

Lambda itself is simple; the architectural decisions that matter surround it. The most common anti-pattern in new serverless designs is orchestration logic — sequencing, retry policy, conditional branching, waiting on a human approval — hand-coded inside the function body instead of expressed in Step Functions. When that logic lives in code, the workflow can’t be visualized, a single failed step can’t be retried independently, and any change to the workflow requires a full function redeploy. Step Functions turns the workflow itself into a reviewable, testable, versioned artifact (a state machine definition) rather than an implicit set of code paths.

EventBridge and decoupling

Direct Lambda-to-Lambda invocation feels simple initially but tightly couples the caller to the callee’s availability and creates synchronous failure propagation — if the downstream function degrades, the upstream one degrades with it. EventBridge (or SNS/SQS, depending on the required fan-out shape) decouples producer from consumer: the producer publishes an event and returns immediately, new consumers can subscribe without the producer being modified or even aware of them, and a failing consumer does not take down the producer. EventBridge additionally supports content-based filtering rules and schema discovery, letting consumers subscribe to a narrow slice of event types without the producer needing to know about that filtering.

API Gateway: REST vs. HTTP APIs

API Gateway is the ingress layer. REST APIs (v1) carry features — usage plans, request/response validation, native WAF integration, private VPC endpoints — that many teams pay for without using. For a straightforward Lambda-proxy pattern, HTTP APIs (v2) are usually meaningfully cheaper and have lower per-request latency overhead. The default should be HTTP API unless a specific v1-only feature is genuinely required by the design.

Lambda Destinations

For asynchronous invocations, a failure can occur before the function’s own error-handling code ever executes — throttling, a permissions error, an internal service exception. Destinations are configured outside the function and route based on the invocation’s actual success/failure outcome as reported by the Lambda service itself, catching failure classes that an in-code try/catch structurally cannot see. This is a core building block for reliable async processing without resorting to polling for failure state. Destinations can route to another Lambda function, SQS, SNS, or EventBridge, and can be configured separately for on-success and on-failure outcomes.

Event-driven design discipline

Building reliable event-driven systems requires deliberately addressing:

• Idempotency — a message may be delivered more than once; processing it a second time must be safe (e.g., via an idempotency key checked against a store before applying an effect).
• Ordering — strict ordering (FIFO queues/topics) costs throughput; it should be a deliberate requirement, not a default, since most event-driven use cases don’t actually need it.
• Dead-letter handling — a message that exhausts all retries must land somewhere visible (a dead-letter queue) rather than disappearing silently. The alternative is a production incident discovered by a customer instead of a dashboard.

None of these are optional hardening steps to add later — skipping them is a latent bug, not a shortcut.

Cost optimization in serverless

Unlike EC2, there is no idle-capacity waste to hunt for in serverless — compute is billed per invocation/duration. The waste that does accumulate is almost always one of two things: Lambda memory left at whatever value made the function “work” initially and never right-sized afterward, or orchestration/request volume nobody is tracking (API Gateway requests, EventBridge events, Step Functions state transitions) until the bill arrives as a surprise. Standard Step Functions bills per state transition; Express Workflows bill per invocation/duration/memory instead and are typically far cheaper for high-volume, short-duration workflows, at the cost of a shorter execution history retention window and at-least-once (rather than exactly-once) semantics for Standard’s default.

REST API vs. HTTP API (API Gateway)

Standard vs. Express Step Functions

Direct Invocation vs. EventBridge/SNS/SQS

Command / Configuration Reference

# Configure a Lambda Destination for async invocation failures
aws lambda put-function-event-invoke-config \
  --function-name my-function \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:123456789012:dlq"}}'

# Enable Provisioned Concurrency on a specific published version
aws lambda put-provisioned-concurrency-config \
  --function-name my-function \
  --qualifier 1 \
  --provisioned-concurrent-executions 5

// EventBridge rule: route only "OrderCreated" events to a target
{
  "Source": ["com.myapp.orders"],
  "DetailType": ["OrderCreated"]
}

# SQS redrive policy: dead-letter after 5 failed processing attempts
RedrivePolicy:
  deadLetterTargetArn: !GetAtt DLQ.Arn
  maxReceiveCount: 5

// Step Functions: Retry with exponential backoff and jitter
{
  "Retry": [
    {
      "ErrorEquals": ["States.TaskFailed"],
      "IntervalSeconds": 2,
      "MaxAttempts": 5,
      "BackoffRate": 2.0,
      "JitterStrategy": "FULL"
    }
  ]
}

Common Pitfalls

• Pitfall: Orchestration logic (retries, branching, sequencing) hand-coded inside Lambda functions. Why: The workflow has no single reviewable representation and every change requires a full redeploy. Fix: Express the workflow in Step Functions as a versioned, testable state machine.
• Pitfall: Services invoking each other directly via synchronous Lambda calls. Why: This creates tight coupling and synchronous failure propagation between components that should fail independently. Fix: Route communication through EventBridge, SNS, or SQS to decouple producer from consumer.
• Pitfall: No dead-letter queue configured on async or queue-based processing. Why: A message that exhausts retries disappears silently instead of surfacing for investigation. Fix: Configure a DLQ (or EventBridge/Lambda Destination on-failure route) on every async processing path.
• Pitfall: Treating “add a retry loop” as a complete fix for a flaky downstream dependency. Why: Naive retries without exponential backoff and jitter can create a retry storm that amplifies load on an already-struggling dependency. Fix: Use exponential backoff with jitter, cap max attempts, and pair it with a dead-letter destination for exhausted retries.
• Pitfall: Non-idempotent consumers processing redelivered messages. Why: At-least-once delivery is the default across SQS, SNS, and EventBridge, so duplicate delivery is a “when,” not an “if.” Fix: Design consumers to be idempotent (e.g., check an idempotency key before applying an effect).
• Pitfall: Using REST API (v1) for a simple Lambda-proxy pattern. Why: REST APIs carry usage-plan, validation, and WAF features that add cost most simple proxy use cases don’t need. Fix: Default to HTTP API (v2) unless a specific v1-only feature is required.

Interview Questions

• “Design a 5-step order fulfillment workflow with retries and a manual approval step. Where does each piece of logic live, and why?” — tests whether Step Functions is the instinct for orchestration, not hand-coded control flow.
• “A consumer service processed the same payment event twice and double-charged a customer. Walk me through the fix.” — tests idempotency understanding, a common real-world failure mode.
• “When would you choose Express Workflows over Standard Workflows in Step Functions, and what do you give up?” — tests understanding of the cost/duration/execution-history/semantics trade-off, not just that both exist.
• “API Gateway costs are climbing faster than traffic growth explains. What do you check first?” — tests whether REST-vs-HTTP API type is considered before assuming volume is the only cost driver.
• “Why might a Lambda function’s own try/catch fail to catch every failure in an asynchronous invocation?” — tests understanding of pre-execution failures (throttling, permissions) and why Lambda Destinations exist.