24. Performance Engineering
Performance Engineering
TL;DR
- Measure the actual bottleneck before optimizing; optimization without measurement is wasted effort.
- Caching is 50% deciding what to cache, 50% invalidating cached data; stale-data bugs indicate invalidation design failure.
- CloudFront caches at edge for shared content (static assets, cacheable API responses); ElastiCache caches at application tier for user-specific or frequently-mutated data.
- Performance metrics: always measure tail latency (p95, p99, p999), not just average or p50. Tail determines user experience.
- Storage optimization: gp3 is the default for most workloads; Provisioned IOPS is justified for consistent, high, predictable IOPS (demanding databases).
Core Concepts
Measurement: The Foundation of Performance Work
Optimizing without measurement guarantees wasted effort. A common scenario: a team spends a sprint on database query optimization, and customer-facing latency barely improves. Root cause: the real bottleneck was a slow third-party API call, not the database.
Disciplined approach:
- Instrument the user-visible path (frontend load time, API latency, database time, network time).
- Measure each component’s contribution.
- Identify the bottleneck (the component consuming the most time).
- Optimize that component.
Example: A page load is 5 seconds total: 1 second to load HTML, 2 seconds to fetch data from API, 1 second to render, 1 second for analytics. The bottleneck is the API (2 seconds). Optimizing the frontend rendering saves 1 second but doesn’t move the overall dial; optimizing the API moves the total from 5 to 4 seconds.
Caching: What and How to Invalidate
Caching is 50% choosing what to cache and 50% invalidating cached data. Every stale-data bug indicates invalidation design failure, not caching mechanism failure.
Cache-aside pattern:
- Check the cache for the data.
- If cache miss, fetch from source.
- Update the cache.
- Serve the data.
Invalidation problem: When the source data changes, the cache still serves the old value. Invalidation must happen on write.
Example: User profile is cached with TTL=1 hour. User updates their name. Application doesn’t invalidate the cache; the cache serves the old name for up to 59 more minutes.
Correct approach: On write (name update), invalidate the cache entry immediately. Or use a shorter TTL for data that changes frequently.
CloudFront vs. ElastiCache
CloudFront: Edge caching. Sits at global edge locations close to end users. Caches HTTP responses. Best for content shared across many users.
Use CloudFront for:
- Static assets (HTML, CSS, JavaScript, images).
- Cacheable API responses (e.g., product catalog, not user-specific data).
- Public content.
ElastiCache: Application-tier caching. Sits close to the application and database. Caches any data structure (strings, objects, lists, sets).
Use ElastiCache for:
- User-specific data (user profile, shopping cart).
- Frequently-queried, frequently-updated data (inventory count, leaderboard).
- Database query results (reducing database load).
Why not use CloudFront for user-specific data: Edge caches should be simple to reason about. Caching user-specific data at global edges creates invalidation complexity (which edge holds which user’s data?) and privacy concerns (edge node geographically far from user holds their data).
Tail Latency: The Real User Experience
Average latency is misleading. A system with average latency of 100ms but occasional 5-second spike is bad for users experiencing the spike.
Percentiles:
- p50 (median): Half of requests are faster; half are slower.
- p95: 95% of requests are faster than this; 5% are slower.
- p99: 99% of requests are faster; 1% are slower.
- p999: 99.9% of requests are faster; 0.1% are slower.
Example: Average latency is 100ms. p50 is 90ms (good). p99 is 5 seconds (bad). Optimizing for average while ignoring tail means 1% of users have a terrible experience.
Causes of tail latency: Cold Lambda starts, garbage collection pauses, slow downstream dependencies, lock contention, occasional slow I/O.
Optimization: Focus on reducing tail latency. Measure p95/p99/p999. Optimize the slow 1% of requests, not the average.
Storage Tiers: gp3 vs. Provisioned IOPS
General Purpose (gp3): The default EBS volume type. 3000 IOPS baseline, burst to higher. Good for most workloads.
Provisioned IOPS: Fixed IOPS ceiling (e.g., 16000 IOPS guaranteed). For workloads that need consistent, high, predictable IOPS.
When gp3 suffices: Most workloads have variable IOPS. gp3’s burst capability handles spikes. Baseline of 3000 IOPS covers most needs.
When Provisioned IOPS is needed: A demanding transactional database (a trading system processing 10K trades/second) needs consistent, high IOPS. gp3’s baseline + burst model is insufficient. Provisioned IOPS guarantees the fixed ceiling.
Trade-off: Provisioned IOPS costs significantly more. Use only if you’ve measured sustained IOPS requirements and gp3 doesn’t meet them. Over-provisioning is wasteful.
Example: A database measures 5000 sustained IOPS. gp3 with its 3000 baseline and burst can handle this. No need for Provisioned IOPS. Paying for 16000 IOPS when 5000 is used is waste.
Comparison Table
| Aspect | CloudFront | ElastiCache | Database Replica |
|---|---|---|---|
| Location | Global edges | Application tier | Database tier |
| Latency | Lowest (geographically closest to user) | Low | Medium |
| Best for | Shared, static content | User-specific, frequently-updated data | Read-heavy workloads |
| Invalidation | TTL-based | Explicit invalidation on write | Replication lag |
| Cost | Per-request + egress | Reserved capacity | EC2 instance cost |
Common Pitfalls
| Pitfall | Why | Fix |
|---|---|---|
| Optimize without measuring | Team optimizes the wrong component, wasting a sprint. | Instrument the user-visible path. Measure each component. Identify the bottleneck. Optimize that. |
| Cache invalidation under-designed | Cache serves stale data. User sees old values after updates. | Design invalidation strategy as seriously as the caching decision. Explicit invalidation on write, or short TTL. |
| Optimize average latency, ignore tail | p50 improves but p99 stays high. 1% of users still have terrible experience. | Always measure and optimize p95/p99/p999, not just average. |
| Storage tier set once at launch, never revisited | gp3 chosen initially; workload grows; sustained IOPS exceed gp3 baseline. Performance suffers. | Measure sustained IOPS regularly. Switch to Provisioned IOPS if warranted. Don’t over-provision. |
| CloudFront used for user-specific data | Invalidation becomes complex (which edge holds which user’s data?). Privacy concern (distant edge holds user data). | Use CloudFront only for shared content. Use ElastiCache for user-specific data. |
Interview Questions
-
A team claims they improved performance by 30% based on average latency, but customer complaints about slowness continue. What do you investigate? — Tests understanding of tail latency (p99) as more important than average.
-
Design a caching strategy for a user profile page that must reflect the user’s own recent updates immediately, while still benefiting from caching. — Tests whether invalidation strategy is designed alongside caching, not as an afterthought.
-
When would Provisioned IOPS EBS actually be worth the added cost over gp3? — Tests ability to reason about a genuine trade-off based on measured IOPS requirements, not defaulting to “more expensive = better.”
-
Your application has a 100ms average latency, but users complain about occasional 5-second slowness. How would you investigate and fix this? — Tests understanding of tail latency diagnostics (cold starts, GC pauses, slow dependencies) separate from average-case optimization.