Incident Management & Blameless Postmortems

TL;DR

• Incident response requires rigid structure: Incident Commander (authority, no coding), Communications Lead (status updates), SME (hands-on debugging).
• Severity levels: SEV-1 (core business broken), SEV-2 (major feature degraded), SEV-3 (minor bug, normal business hours).
• Blameless postmortem culture: human error is never root cause; it’s a symptom of flawed systems. Fix the system, not the person.
• 5 Whys technique drills from symptoms to systemic causes. Example: “database crashed” → “ran out of memory” → “bad query” → “staging didn’t mirror production” → fix: sync production data to staging.
• Action item rot is the biggest postmortem failure. SEV-1 action items must have priority over new feature development.
• Timeline, impact, root cause, and action items are the postmortem artifacts. Distribute to organization; make learnings visible.

Core Concepts

Incident Lifecycle

Detection → Response → Mitigation → Recovery → Postmortem → Learning

Incident Commander (IC)

Single point of authority during incident. Does not debug or code. Coordinates team, makes decisions (“failover to region B”), timeboxes analysis, keeps focus.

Communications Lead

Writes status page updates, customer notifications, executive summaries. Shields IC and SME from distraction. Owner of narrative during incident.

Subject Matter Expert (SME)

Engineer with hands-on knowledge of the failing system. Reads dashboards, logs, applies fixes. Reports status to IC.

Blameless Postmortem Culture

Root cause is never “engineer made a mistake.” Mistakes are symptoms. Root cause is the system that allowed the mistake. Fix the system (automation, tests, reviews).

5 Whys RCA (Root Cause Analysis)

Technique to drill from immediate symptom to systemic cause. Ask “why” five times; each answer reveals the next level. Stops at system deficiency, not human action.

Severity Level Definitions & Response

Level	Definition	Examples	Response
SEV-1	Complete outage of core business flow	Checkout broken, auth down, API unresponsive	Page all on-call immediately; IC takes over; drop everything
SEV-2	Major feature degraded; workarounds exist	Search slow, dashboard errors, subset of users affected	Page on-call engineer; SLA to mitigation: 2 hours
SEV-3	Minor issue, cosmetic bug, performance blip	UI typo, slow batch job, single endpoint latency	Create ticket; handle during business hours

Postmortem Structure

Section	Purpose	Examples
Summary & Impact	Who was affected? How long? Revenue impact?	“30-min outage. 100k users. $50k revenue loss. Stripe payments unavailable.”
Timeline	Chronological events: alert fired, detection, mitigation, recovery	“14:05 alert fires. 14:10 IC page team. 14:22 diagnosis: memory leak. 14:35 restart app. 14:40 traffic normal.”
Root Cause Analysis	System deficiency discovered via 5 Whys	“Load test data doesn’t mirror production scale → query not indexed → full table scan → memory exhaustion”
Action Items	Tickets to prevent recurrence (must prioritize SEV-1 items)	“Sync prod data to staging daily. Add query performance test. Implement auto-restart on OOM.”

5 Whys Example: Database Crash

Level	Question	Answer
Symptom	What happened?	Database ran out of memory and crashed
Why 1	Why did it run out of memory?	A query performed a full table scan instead of using an index
Why 2	Why was there no index?	New microservice deployment included a bad SQL query
Why 3	Why didn’t the load test catch it?	Load testing environment had only 1M rows; production has 100M rows
Why 4	Why doesn’t staging mirror production?	No automated data sanitization or syncing
Why 5 (Root Cause)	Why isn’t this enforced?	Staging refresh is manual; no SLA; low priority in backlog
Fix	What system changes prevent this?	Automate production data sanitization + daily staging sync. Add query performance test to CI. Review all new queries on tables >10M rows.

Incident Response Workflow

INCIDENT STARTS
     ↓
PAGE ON-CALL (severity-based)
     ↓
IC ASSUMES COMMAND
     ↓
ESTABLISH: IC + Comms Lead + SMEs
     ↓
STATUS PAGE: "Investigating..."
     ↓
PARALLEL TRACKS:
  · SME: Debug & apply fixes
  · Comms: Update status page every 15min
  · IC: Timebox analysis, make escalation calls
     ↓
MITIGATION: Fix applied, traffic recovers
     ↓
RESOLUTION: All systems green for 15min
     ↓
STATUS PAGE: "Resolved at HH:MM UTC"
     ↓
IC SCHEDULES: Postmortem for <48 hours
     ↓
POSTMORTEM: Timeline, RCA, action items
     ↓
DISTRIBUTION: Share with org, track action items to completion

Common Pitfalls

Pitfall	Why It Happens	Fix
No IC; everyone debates	Unclear authority structure; engineers argue over strategy. Wastes time, delays mitigation.	Define IC role before incident. IC does not participate in debugging—only coordinates. One voice, one decision.
Timeline incomplete or wrong	Timestamps missed during chaos. Post-mortem later can’t reconstruct sequence.	Use incident chat as single source of truth. Require /timestamp [event] in comments. Auto-parse timestamps.
Blame language in postmortem	“Engineer forgot to run migration.” Shifts culture to blame, shame, defensiveness. Repairs are slow.	Reframe to system: “CI/CD allows code deploy without migrations.” Root cause: automation gap. Action: Add pre-flight check.
Action item rot	Brilliant ideas identified. Backlog piles up. No accountability. Items never addressed.	Enforce policy: SEV-1 action items = production priority. Assign owner. SLA to complete. Track public dashboard. Block new features if SEV-1 items pending.
Lessons not distributed	Postmortem written, filed, forgotten. Same incident repeats in different team.	Publish postmortem to Slack, Wiki, email digest. Monthly postmortem review meeting. Trending analysis: “What are our top 5 RCAs this quarter?”
Postmortem too late	Incident on Monday. Postmortem scheduled for the following Monday. Memory fades.	Postmortem within 48 hours while memory is fresh. Requires discipline. Time it during working hours. Participation > perfection.

Interview Questions

Q — You’re IC for SEV-1 database outage. Two engineers debate: restart primary or failover? 15 min elapsed. What do you do?

— Break analysis paralysis immediately. Set timebox: “We spend 5 more minutes trying restart. If not successful by [time], we failover.” Communicate deadline clearly. At T+5min, make the call—no further debate. Owner of decision is the IC, not consensus. Speed > perfection during incident. Document the decision in incident timeline: “[time] IC decision: proceed with failover. Rationale: restart attempts exceeded SLA tolerance.”

Q — Postmortem says: “Engineer forgot to run database migration before deploy.” Reframe blamelessly.

— Never blame the person. Reframe to system deficiency: “Deployment pipeline allows application code and database migrations to be deployed independently, creating version mismatch. Action item: Implement automated pre-flight check in CI/CD that halts deployment if pending migrations detected.” Shift from human error to system improvement. Ask: Why was Bob able to skip the migration? Automate it away.

Q — You identify 5 action items in postmortem. Budget to complete all in one sprint is zero. What’s your priority?

— Prioritize by SEV level + blast radius + likelihood. SEV-1 action items (prevent recurrence of critical outage) take absolute priority—stop new feature work. Assign owner + deadline. Others (SEV-2, SEV-3) go in backlog with SLA (30, 60, 90 days). Public tracking dashboard: show status of all postmortem items. Don’t let the list grow faster than completion rate. Monthly review: celebrate completed items, escalate stalled ones.

Q — How do you establish blameless culture if engineers fear punishment for mistakes?

— Model it from leadership: first postmortem should be from a senior engineer or manager who made a mistake. Frame it as learning opportunity. In postmortem discussions, redirect blame language immediately: “That’s a system design gap, not a person error. How do we automate this?” Celebrate engineers who identify gaps in postmortems; don’t punish them. Punishment drives incidents underground; blameless culture brings them to light for learning.