34. Security Operations
Security Operations
TL;DR
- Generating security findings (GuardDuty, Config, Inspector) is straightforward; what separates mature security operations from a checkbox exercise is triage, response speed, and auditability after a finding fires.
- Findings need severity normalization — an active credential-exfiltration finding and an unused security-group-rule finding cannot sit in the same undifferentiated queue.
- Security Hub (or equivalent) centralizes severity normalization, ownership routing, and compliance-standard mapping across all finding sources.
- EventBridge-driven automated remediation closes well-understood, high-confidence findings in seconds — but blanket remediation rules break legitimate configurations without an exception path.
- Runbooks decay: a stale reference (dead Slack channel, deprecated tool) costs critical time exactly when an incident is in progress.
Core Concepts
Finding severity and triage
GuardDuty, Config, and Inspector findings arrive with genuinely different urgency — active credential exfiltration versus an unused, overly-permissive security group rule are not the same emergency. If severity isn’t actively normalized and triaged, both can land in the same undifferentiated queue. The operational risk: a genuinely urgent finding sits behind a backlog of routine hygiene issues, receiving the same slow handling instead of the fast response its actual risk level demands.
Centralized aggregation (Security Hub)
Beyond simply aggregating findings, Security Hub (or an equivalent aggregation layer) is where severity normalization, ownership routing, and compliance-standard mapping happen centrally. Without this layer, an analyst investigating an incident must manually check multiple separate consoles to piece together the full picture, and there is no consistent place to enforce “findings of this severity get this response SLA” across every detection source.
Automated remediation (EventBridge)
EventBridge rules routing well-understood, high-confidence finding types to automated remediation (typically a Lambda function invoked by the rule) close the gap in seconds instead of the hours a human alert-triage-act cycle takes. Examples: a publicly-exposed S3 bucket that should never be public, an unencrypted volume created against policy.
The design detail that determines whether this is safe: a blanket “make it private” remediation applied to every “publicly exposed” finding will incorrectly break a legitimately public resource — a static website bucket, a public download location. Automated remediation needs an exception path, most commonly a tag-based allow-list, to distinguish intentional public configuration from accidental exposure. A rule that fires identically regardless of intent is a liability, not a safeguard.
Automation should not fire autonomously on ambiguous or high-impact findings, where a wrong automated action could cause its own damage — locking out a legitimate user, disabling a resource mid-use. Those cases stay with human incident command: automation handles the well-understood cases fast, humans handle the judgment calls.
Incident response runbooks
Incident response depends on runbooks that are actually current. A runbook referencing a Slack channel that no longer exists, or a deprecated tool, fails exactly when it matters most — during a real incident, under time pressure, when a broken reference costs response time nobody has to spare. Runbook staleness is a gradeable finding in a security operations review, not a minor documentation nitpick; runbooks require the same maintenance discipline as the infrastructure and processes they describe.
Automated Remediation vs Human-in-the-Loop Response
| Aspect | Automated Remediation | Human-in-the-Loop |
|---|---|---|
| Best suited for | Well-understood, high-confidence, low-collateral-damage findings (public bucket, unencrypted volume) | Ambiguous or high-impact findings where a wrong action causes its own damage |
| Response time | Seconds (Lambda triggered by EventBridge rule) | Minutes to hours (alert, triage, manual action) |
| Failure mode if misapplied | Blanket rule breaks a legitimate configuration (e.g., intentionally public static site) | Slow response lets a genuinely urgent finding sit unaddressed |
| Requires | Exception path (tag-based allow-list) to distinguish intent | Clear escalation path, current runbooks, defined ownership |
Common Pitfalls
- Pitfall: Findings handled in raw arrival order with no severity normalization. Why: All findings land in one undifferentiated queue regardless of actual risk. Fix: Apply Security Hub (or equivalent) severity normalization and route by risk, not arrival time.
- Pitfall: Automated remediation with no exception path. Why: A blanket rule can’t distinguish intentional public configuration from accidental exposure. Fix: Add a tag-based allow-list or similar exception mechanism before automating remediation.
- Pitfall: Everything automated with no human review layer. Why: Ambiguous or high-impact findings can trigger an automated action that causes its own incident. Fix: Reserve automation for well-understood, low-collateral-damage cases; route ambiguous cases to human review.
- Pitfall: Incident response runbooks that were accurate when written but never revisited. Why: Tools, channels, and contacts change over time; the runbook doesn’t track those changes automatically. Fix: Maintain runbooks with the same discipline as the infrastructure they describe — scheduled review, not one-time authorship.
- Pitfall: Analysts manually checking multiple security consoles during an investigation. Why: No central aggregation layer was ever built. Fix: Stand up a single aggregation point (Security Hub or equivalent) for cross-service finding visibility.
Interview Questions
- “Design automated remediation for publicly-exposed S3 buckets that won’t break a legitimately public static website bucket.” — tests whether the exception-path design detail is understood, not just the automation concept.
- “Your team has GuardDuty, Config, and Inspector all enabled, but analysts still check three separate consoles during an incident. What’s missing?” — tests whether centralized aggregation is understood as an operational necessity.
- “When would you deliberately choose NOT to automate remediation for a security finding, even though you technically could?” — tests judgment about where human review still matters versus where automation is safe.
- “How would you design a severity-triage process so an active credential-exfiltration finding never sits behind routine hygiene findings?” — tests understanding of severity normalization and SLA-based routing.