Observability Platform

TL;DR

• Metrics (CloudWatch), logs (CloudWatch Logs), audit trail (CloudTrail), and configuration history (Config) each answer a distinct question — none substitutes for another.
• CloudTrail answers “who did what, when” at the AWS API level; Config answers “what did this resource’s configuration look like at a point in time.” Conflating the two is a common, testable gap.
• X-Ray provides per-hop latency across a distributed request path — aggregated per-service metrics can’t show a single request’s journey.
• AWS Health Dashboard surfaces AWS-side operational issues that internal monitoring cannot see, since the problem originates in the platform, not the application.
• Dashboards require a human to look at the right time; real observability maturity means alerts fire proactively on golden signals with clear ownership.
• Centralized logging across accounts is what makes cross-account incident correlation and consistent retention/access control possible.

Core Concepts

Metrics vs logs

CloudWatch (metrics) answers “what is the current and historical numeric state of this system” — CPU, request count, custom application metrics. CloudWatch Logs answers “what did the application or system actually record happening.” Conflating the two is a common early mistake: metrics indicate that something is wrong, logs reveal what and often why.

CloudTrail: the audit trail

Answers a specifically security-relevant question: who did what, when, from where, using which credentials, at the AWS API level. If an incident review needs to know who deleted a resource or changed a security group, CloudTrail is the source of truth. CloudWatch alone is not built to answer that question. Incident reviews stall when CloudTrail was not enabled organization-wide from the start — audit history cannot be retrofitted after the fact.

AWS Config: configuration state history

Answers a distinct question from CloudTrail: what did a resource’s configuration actually look like at a specific point in time. CloudTrail shows that a change happened; Config gives the historical configuration state itself and evaluates compliance rules against that history.

X-Ray: distributed tracing

Solves the problem of a single request crossing multiple service boundaries, where per-hop latency is needed to isolate the actual bottleneck. Aggregated per-service CloudWatch metrics show that a service’s average latency rose, but not one specific slow request’s journey through services A→B→C→D. Past a certain microservice complexity threshold, distributed tracing stops being optional.

EventBridge in an observability role

Routes operational events — a GuardDuty finding, a Config compliance change, an AWS Health Dashboard notification — to an automated response instead of a dashboard nobody is watching in real time.

AWS Health Dashboard

Surfaces AWS-side operational issues (a Region having a bad day, a maintenance event affecting specific resources) invisible to internal monitoring because the problem originates in the underlying platform, not the application code. Teams without Health Dashboard wired into alerting frequently spend significant time debugging their own stack for a problem that is actually AWS-side and already acknowledged.

Centralized logging

Aggregating logs from all accounts into a dedicated log archive account (via CloudWatch Logs subscription filters, Kinesis Data Firehose, or an equivalent pipeline) turns per-account visibility into organization-wide incident correlation, with consistent retention policies and access controls.

CloudTrail vs Config vs CloudWatch vs X-Ray

Common Pitfalls

• Pitfall: CloudTrail not enabled organization-wide until after an incident. Why: Audit trail setup is deprioritized until it’s needed. Fix: Enable CloudTrail via Control Tower or an Organizations trail from day one — audit history cannot be retrofitted.
• Pitfall: Confusing CloudTrail (change history) with Config (configuration state history). Why: Both relate to “what changed,” but answer different questions. Fix: Use CloudTrail to find who made a change; use Config to see exact historical configuration state.
• Pitfall: Dashboards exist but nobody is actively alerted from them. Why: A dashboard is passive — it requires a human to look at the right time. Fix: Tie alerting to golden signals (latency, traffic, errors, saturation) with clear, tested on-call ownership.
• Pitfall: No distributed tracing on a genuinely multi-service architecture. Why: Tracing is added reactively only after a hard-to-diagnose latency incident. Fix: Wire in X-Ray (or equivalent) before it’s needed, as request paths cross more than a couple of services.
• Pitfall: Debugging an application issue for an extended period before discovering it was an AWS-side event. Why: AWS Health Dashboard is not integrated into the alerting pipeline. Fix: Feed Health Dashboard events into the same alerting path as application-level alerts.

Interview Questions

• Walk through the exact difference between what CloudTrail and AWS Config each tell you, with a concrete example of when each is needed. — tests precise understanding, not just name recognition.
• A customer reports intermittent slowness across a 6-service request path. Walk through the diagnostic approach. — tests whether X-Ray (or equivalent tracing) is the instinct rather than manual log correlation.
• Design centralized logging for a 40-account AWS Organization, including retention and access control. — tests real architectural thinking about the log archive pattern, not just “enable CloudWatch everywhere.”
• A security incident review needs to know exactly who deleted a production S3 bucket. Which service answers that, and why wouldn’t CloudWatch alone suffice? — tests whether the audit-trail role of CloudTrail is understood distinctly from metrics/logs.