Terraform

Terraform is an infrastructure-as-code tool: you declare the desired end state of infrastructure in HCL, and Terraform computes and executes the API calls to get there.

Declarative vs imperative scripting:

• A script says do these steps — run it twice and you get errors or duplicates; handling 'already exists' is your problem
• Terraform says this should exist — it compares desired state against known state and generates only the diff: create what's missing, update what changed, destroy what was removed. Running it twice is a no-op (idempotency for free)

The engine behind it:

1. Providers translate HCL resources into API calls (AWS, GitHub, Datadog — 3,000+)
2. State records what Terraform created, mapping config to real-world IDs
3. The graph — resources form a dependency DAG; independent resources are applied in parallel, dependents in order

One-liner: 'scripts encode how, Terraform encodes what — and the diff engine turns what into a reviewable execution plan.'

provider "aws" {            # HOW to talk to an API
  region = "us-east-1"
}

resource "aws_instance" "api" {   # something Terraform OWNS
  ami           = data.aws_ami.al2023.id
  instance_type = "m5.large"
}

data "aws_ami" "al2023" {         # something Terraform READS
  most_recent = true
  owners      = ["amazon"]
  filter { name = "name"; values = ["al2023-ami-*"] }
}

• Provider — the plugin that knows an API: auth, endpoints, resource schemas. Configured once, versioned explicitly
• Resource — infrastructure Terraform creates, updates, and destroys. It lives in state; deleting the block deletes the thing
• Data source — a read-only query against existing infrastructure (someone else's VPC, the latest AMI, an existing zone). Never modified, never destroyed — just fetched at plan/apply time

The interview distinction: resource = ownership and lifecycle; data source = reference without ownership. Mixing them up — importing shared infra as a resource in five different stacks — is how one team's terraform destroy deletes another team's VPC.

State (terraform.tfstate) is Terraform's database: a JSON record mapping each config block to the real resource it created (aws_instance.api → i-0abc123), plus all known attributes and dependency metadata.

Why it must exist:

1. Identity mapping — clouds don't tag resources as 'created by this config block'; without state, Terraform can't know aws_instance.api is i-0abc123 vs an identical instance someone else made
2. Diffing without full API scans — plan compares config ↔ state (and refreshes state ↔ reality); no state means interrogating every API for every possible resource
3. Tracking the unknowable — some values exist only at creation time (generated passwords, IDs); state remembers them
4. Dependency ordering for destroy — config says what should exist; only state knows what does exist and in what dependency order to unwind it

Operational consequences that follow:

• State is a crown jewel: it contains secrets in plaintext (DB passwords, keys) — encrypt it, restrict access, never commit it to git
• Losing state doesn't delete infrastructure — it deletes Terraform's knowledge of it (recovery = re-import, painful)
• Two people applying with different state copies = chaos — hence remote backends with locking

• terraform init — one-time setup per directory: downloads providers (pinned via .terraform.lock.hcl — commit it), fetches modules, configures the backend (state location). Safe to re-run; required after adding providers/modules/backend changes
• terraform plan — the dry run: refreshes state against reality (API reads), diffs desired (config) vs actual, prints the execution plan: + create, ~ update in-place, -/+ destroy-and-recreate (the one to read twice), - destroy. -out=plan.tfpb saves the exact plan for apply
• terraform apply — executes the plan (re-plans first unless given a saved plan file), walks the dependency graph in parallel (default 10 concurrent), updates state as it goes. CI best practice: apply the saved plan file — guarantees what was reviewed is what runs, no TOCTOU gap
• terraform destroy — plans and executes deletion of everything in state, reverse dependency order. In practice: gated, rare, and often replaced by deleting config + applying

The habit that marks production experience: nobody applies unreviewed plans to prod — plan output is a reviewed artifact (PR comment), and -/+ replacements on stateful resources get called out explicitly before anyone types yes.

variable "environment" {          # INPUT — caller supplies
  type        = string
  description = "Deploy environment"
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Must be dev, staging, or prod."
  }
}

locals {                          # COMPUTED — internal only
  name_prefix = "payments-${var.environment}"
  common_tags = { env = var.environment, team = "payments" }
}

output "alb_dns" {                # EXPORT — for humans & other stacks
  value       = aws_lb.main.dns_name
  description = "Public ALB endpoint"
}

• Variables — the module/stack's input API: typed (string, number, list, map, object), validated, documented. Supplied via terraform.tfvars, -var, TF_VAR_* env vars. sensitive = true redacts from output (not from state!)
• Locals — named intermediate expressions: computed values, DRY for repeated expressions, assembling tags/names. Not settable from outside
• Outputs — exported values: consumed by humans, by parent modules (module.vpc.vpc_id), or by other stacks (via terraform_remote_state or better, data sources)

Design smell to name: a module with 60 variables is a config file wearing a module costume — inputs should be a deliberate, minimal API.

A module is a reusable unit of Terraform config — a directory of .tf files with typed inputs (variables) and outputs. Every directory you run terraform in is already the 'root module'; child modules are called with:

module "vpc" {
  source  = "git::https://github.com/corp/tf-modules//vpc?ref=v2.3.0"
  cidr    = "10.20.0.0/16"
  az_count = 3
}

What makes a good module:

1. One purposeful abstraction — 'a VPC with our standards', 'a service's full serving stack' — not 'a thin wrapper around one resource' (adds indirection, subtracts nothing) and not 'our entire platform' (untestable monolith)
2. Minimal input API, strong defaults — callers set what varies; org standards (tags, encryption, logging) are baked in, not parameters
3. Versioned sources — git tags or a registry with semver; ref=main in prod is a time bomb
4. Composable — outputs expose what downstream needs (IDs, ARNs); modules take IDs in, not assumptions about where they come from
5. Documented + tested (examples/ directory that CI actually applies)

One-liner: 'a module is an API for infrastructure — design the interface first, the resources are implementation.'

The backend determines where state lives and how it's locked. Default is a local file — fine for one laptop, broken for a team (no sharing, no locking, no durability).

The standard AWS backend:

terraform {
  backend "s3" {
    bucket       = "corp-tfstate-prod"
    key          = "network/vpc/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # S3-native locking (modern; DynamoDB table was the classic)
  }
}

What each piece buys:

• S3 — durable, versioned (enable bucket versioning — it's your state recovery story), encrypted at rest (SSE-KMS), access-controlled via IAM
• Locking — prevents two concurrent applies corrupting state: historically a DynamoDB table (LockID), now S3-native lockfiles do it without the extra table
• Key-per-stack — the object key namespaces many states in one bucket

Hardening expected in the answer: bucket versioning ON (point-in-time state recovery), strict IAM (state contains secrets — read access is secret access), KMS CMK, access logging, and MFA-delete/object-lock for the truly cautious.

Bonus point: state access is a bigger deal than people think — CI roles with state-read can exfiltrate every secret Terraform ever touched.

Both create multiple instances from one block:

# count — positional, indexed
resource "aws_subnet" "private" {
  count      = 3
  cidr_block = cidrsubnet(var.cidr, 4, count.index)
}

# for_each — keyed by map/set
resource "aws_subnet" "private" {
  for_each   = { a = "10.0.1.0/24", b = "10.0.2.0/24", c = "10.0.3.0/24" }
  cidr_block = each.value
}

The count footgun (a guaranteed interview follow-up): count instances are tracked by index — aws_subnet.private[0], [1], [2]. Remove the first element of the list feeding count, and every subsequent resource shifts index → Terraform plans to destroy and recreate all of them for what was logically a one-item removal. With subnets or databases, that plan is an outage.

for_each tracks by key — aws_subnet.private["a"] — removing "b" touches only "b". Stable identity under change.

The rules:

• Collections of named things that evolve → for_each, always
• Pure replication where instances are interchangeable and only the number matters (or conditional creation via count = var.enabled ? 1 : 0) → count is fine
• Migrating between them rewrites state addresses — use moved blocks or terraform state mv, not destroy/recreate

The uncomfortable truth first: anything Terraform knows ends up in state in plaintext — sensitive = true only redacts CLI output. A generated RDS password, a created API key: all readable by anyone with state access. So secret strategy = state-access strategy.

The patterns, best first:

1. Don't make Terraform know the secret at all — create the container, let something else set the value: Terraform creates the Secrets Manager secret (name, KMS key, IAM); the value is set by a rotation Lambda, an admin one-time, or the service itself. State holds only ARNs
2. Native generation with managed rotation — e.g. manage_master_user_password on RDS: AWS generates and rotates the password in Secrets Manager; Terraform never sees it
3. Reference at deploy-time, not TF-time — apps get secret ARNs/names from Terraform outputs and fetch values at runtime (IRSA/task roles) — infra code and secret values never meet
4. When Terraform must read a secret (data source from Vault/SSM to configure a provider): accept it's in state; scope that stack's state tightly, and prefer ephemeral values / short-lived creds where supported

Hygiene floor: encrypted backend (KMS), IAM on state = secret-grade, no .tfstate or .tfvars with secrets in git (pre-commit scanning), CI masks output, and rotation invalidates whatever state has seen.

One-liner: 'treat state as a secrets file that happens to contain infrastructure — then design so the interesting secrets never enter it.'

Terraform — declarative HCL, provider ecosystem across every API (cloud, SaaS, K8s), explicit state you own, huge hiring pool and module registry.

• Weaknesses: state is your operational burden; HCL logic beyond moderate complexity gets contorted (though functions/dynamic blocks cover most needs)

CloudFormation — AWS-native: state managed by AWS (stacks), deep IAM integration, StackSets for org-wide rollout, drift detection built in.

• Weaknesses: AWS-only (multi-cloud and SaaS providers don't exist), slower feature lag for new services vs the AWS TF provider, verbose templates, painful rollback debugging at scale

Pulumi / CDK — real languages (TypeScript, Python, Go): loops, types, tests, abstractions from software engineering. CDK compiles to CloudFormation (inherits its ceiling); Pulumi has its own engine + state service.

• Weaknesses: 'it's real code' cuts both ways — undisciplined teams build clever unmaintainable infra; smaller talent pools; CDK ties you to CFN semantics

How I actually decide:

• Default: Terraform — ecosystem breadth (you will need the Datadog/GitHub/K8s providers), org-scale patterns are well-trodden, hiring is easy
• All-in AWS shop with strong AWS support relationship and no multi-provider needs → CloudFormation/CDK is defensible
• Platform teams building rich abstractions for developers, TypeScript-native org → Pulumi shines
• The non-negotiable: pick ONE as the org standard — the worst outcome is three tools with three half-owned states describing overlapping infrastructure

One-liner: 'the tool matters less than the discipline around it — but Terraform's provider ecosystem is the moat the others haven't crossed.'

State surgery is how you change Terraform's bookkeeping without touching real infrastructure:

• terraform state mv — rename/move an address: refactoring (aws_instance.web → module.web.aws_instance.this), moving resources between modules. Wrong-address symptoms: plan wants to destroy X and create identical Y — that's a rename Terraform didn't see
• moved blocks (the modern way): declare the rename in code — moved { from = aws_instance.web; to = module.web.aws_instance.this } — applied automatically, reviewable in the PR, works for everyone who runs the config (state mv is imperative, per-state, easy to forget on one of 12 workspaces). Refactors should ship moved blocks, not runbooks
• terraform import / import blocks — adopt existing infrastructure into state: import { to = aws_s3_bucket.logs; id = "corp-logs" } + write matching config. Import blocks (1.5+) beat the CLI: plannable (plan shows what import will do), batchable, reviewable, and -generate-config-out drafts the HCL for you
• terraform state rm — forget a resource without destroying it: handing ownership to another stack/tool, or excising something manually deleted that refresh can't reconcile. The paired danger: config still present + state rm = Terraform now wants to create a duplicate

The discipline that separates seniors: every surgery is preceded by a state backup (terraform state pull > backup.json — versioned S3 gives you this anyway) and followed by a zero-diff plan as proof of correctness. Surgery that ends with 'plan shows changes' isn't done.

One-liner: 'plan-visible destroy/create pairs on unchanged infra are almost always identity problems — fix the bookkeeping with moved/import, never let Terraform "fix" it with a replace.'

The core tension: one big state = one lock, one blast radius, 20-minute plans, everyone blocked on everyone. Ten thousand tiny states = orchestration hell and cross-references everywhere. Design for blast radius, ownership, and change cadence.

The split I use:

Per account/environment:
├── network/          # VPC, TGW attachments, DNS — changes quarterly, platform-owned
├── security/         # IAM baseline, KMS, org guardrails — high privilege, tightly gated
├── data/<system>/    # RDS, DynamoDB — stateful, careful cadence, service-team owned
└── services/<team>/  # per-team app infra — changes daily, team-owned

The rules behind it:

1. Split on change cadence — daily-change app infra never shares state with quarterly-change networking; a bad refactor in a service stack must not be able to plan a VPC destroy
2. Split on ownership — state boundary = IAM boundary: teams get apply rights on their states only; the security stack needs different humans approving
3. Split on privilege — the stack that manages IAM is its own state with its own pipeline gates; 'app deploy pipeline can modify org IAM' is a finding, not a convenience
4. Cross-state references flow one way — services read network outputs (via data sources on resource attributes, preferred over terraform_remote_state which couples you to state layout and grants state-read = secret-read); never circular

Blast-radius math to say out loud: the worst-case terraform destroy (or malicious plan) in any single state should be one team's redeployable infrastructure — never data, never the network, never identity. If a single state's destruction would take >1 team down for >1 day, split it.

Scale mechanics: consistent per-stack pipelines (same plan/apply workflow stamped everywhere), state key naming convention as code, and a stacks inventory (who owns what, last applied when) — 400 states without an inventory is archaeology.

One-liner: 'state boundaries are organizational and security boundaries wearing an infrastructure costume — draw them where ownership, privilege, and cadence change.'

What workspaces are: one configuration, N named states (terraform workspace select prod) — same code, switched state files, terraform.workspace available for conditionals.

Why directory-per-env wins for production:

1. Environments differ by more than state — different backend configs (prod state in the locked-down bucket), different provider configs (different accounts/roles), different versions during rollouts. Workspaces share ALL of that — one backend, one privilege context, one code version, always
2. The conditional rot: workspace-based configs sprout count = terraform.workspace == "prod" ? 3 : 1 everywhere — environment differences become invisible in review (what changes in prod? grep for workspace conditionals and simulate in your head) instead of a diffable prod/main.tf or tfvars file
3. Promotion is opaque: with directories, promoting = a PR diff between env configs (reviewable); with workspaces, both envs always run the same commit — you can't hold prod back while staging soaks a change without branch gymnastics
4. Wrong-workspace accidents: terraform destroy with a forgotten workspace select is a genuine incident class — directories make the target physically explicit (you're standing in prod/)
5. IAM separation: you can't give different humans different rights to different workspaces of one backend key; directory + per-env backend = per-env IAM

The directory pattern done right (no code duplication): envs are thin instantiation shells — envs/prod/main.tf is ~20 lines calling shared modules with prod values + prod backend. Logic lives in modules once; environments are parameterizations. (This is exactly the layout Terragrunt formalizes.)

Where workspaces are legitimately fine: ephemeral same-shape instances — preview environments per PR, developer sandboxes — same code, same account, same privileges, disposable state. That's the use case they model well.

One-liner: 'workspaces multiplex state; environments differ in code, config, credentials, and cadence — use workspaces for clones, directories for environments.'

The flow:

PR opened → fmt/validate/lint → plan → policy checks → plan posted to PR
         → human review (the PLAN is the review artifact, not just the HCL)
merge    → apply the SAVED plan → verify → notify

Stage by stage:

1. Static (seconds): terraform fmt -check, validate, tflint (provider-aware: catches invalid instance types, deprecated arguments), checkov/trivy for misconfig scanning (public S3, open SGs, unencrypted anything)
2. Plan (per changed stack): detect changed stacks (path filtering / dependency graph), run terraform plan -out=plan.bin with a read-only role — plan needs describe*, never write. Post rendered plan as PR comment; large plans get a summary (X create, Y update, Z destroy — listed explicitly)
3. Policy as code: OPA/Sentinel/Conftest against the plan JSON (terraform show -json plan.bin): deny public ingress, require tags, block -/+ on protected resource types without a break-glass label, cost gate via Infracost (delta > $500/mo needs lead approval — posted on the PR)
4. Apply on merge: apply the exact saved plan artifact (terraform apply plan.bin) — what was reviewed is what runs; if the plan is stale (state changed since), apply fails safe and re-plans for re-review. Prod applies: separate runner with the write role (OIDC federation, no static keys), serialized per stack (locking makes this natural), during defined windows for sensitive stacks
5. Post-apply: zero-diff verification plan, output diffs to the PR, drift detection continues on schedule (nightly plan across all stacks → alerts on unexpected diffs)

The security details that get probed:

• Plan-stage code execution: terraform plan executes provider code and external data sources — a malicious PR can exfiltrate from the plan runner. So: plan role is read-only, plan runners are isolated per-PR, and external data sources/providers are allowlisted
• Fork PRs never get credentials; state access split (plan role reads state, apply role writes)

Tooling honesty: this is exactly what Atlantis/TFC/Spacelift/env0 productize — build vs buy depends on stack count and compliance needs; the shape above stays the same.

One-liner: 'the unit of review is the plan, the unit of execution is the saved plan artifact, and the plan stage is untrusted code execution — design all three accordingly.'

lifecycle {
  prevent_destroy       = true
  ignore_changes        = [tags["LastScanned"], desired_count]
  create_before_destroy = true
  replace_triggered_by  = [aws_launch_template.app]
}

prevent_destroy — plan-time hard failure on any destroy of this resource.

• Prevents: the classic 'refactor renamed the resource, plan quietly included destroying the production database, reviewer skimmed' incident. Belongs on: databases, state buckets, KMS keys, anything whose loss is data loss
• Causes: friction during intentional rebuilds (must edit code to remove it — which is the point); doesn't survive the resource being removed from config entirely (orphan-then-manual-delete still possible) — it's a seatbelt, not a cage

ignore_changes — refresh sees drift on these attributes; plan pretends it doesn't.

• Prevents: ownership fights — ECS desired_count managed by autoscaling, tags stamped by external systems, fields mutated by lambdas/operators. Without it: every apply reverts the autoscaler and you get the 2am 'apply scaled us down' incident
• Causes: silent config rot — the ignored attribute in code drifts ever further from reality; new environments built from that code get the stale value (dev has the old instance size 'because ignore_changes'). Rule: every ignore_changes entry needs a comment naming the other owner, and ignore_changes = all is a code smell bordering on incident report

create_before_destroy — inverts replacement order: build new, then remove old.

• Prevents: replacement downtime — new launch template instances/certs/SGs come up before the old die; essential for anything serving traffic where the default destroy-first ordering is an outage
• Causes: name-collision failures (can't create the new one because the name is taken — use name_prefix/generated names) and transient double-capacity (quota headroom needed). Also propagates: CBD on a resource forces CBD-compatible ordering on its dependencies

replace_triggered_by — force replacement when a referenced resource changes; the clean way to tie instance refresh to template changes without taint hacks.

One-liner: 'lifecycle args encode operational truths — what must never die, what someone else owns, what can't have a gap. Each one is also a small lie to the planner, so each one earns a comment.'

Situation: a PR renaming/retagging infra shows -/+ aws_db_instance.main (forces replacement) buried in a 40-resource plan. This is the Terraform near-miss scenario — plan-approved replacements on stateful resources are how databases get deleted by CI.

Immediate response:

1. Stop the line — plan does not merge; nothing time-critical about it. (This is why destroy/replace counts are surfaced loudly in the PR comment and why policy gates flag -/+ on protected types)
2. Find the forcing attribute: the plan marks it — # forces replacement on the specific field. Common culprits: identifier rename, engine edits, subnet/AZ changes, immutable storage params, or a module refactor changing a computed name
3. Decide: avoid, or orchestrate

Avoiding the replacement (usual path):

• Rename-driven? Revert the name, or if the address changed, moved block / state mv — bookkeeping fix, zero-diff plan proves it
• Attribute genuinely needs changing but is replacement-forcing? Check for an in-place path: many 'immutable' things have blue-green alternatives (RDS: modify + apply-immediately windows, or blue/green deployments feature) — the provider's replacement isn't the only migration

If replacement is truly intended (engine major, storage rearchitecture): it becomes a migration project, not a terraform apply: snapshot + verified restore path, replica promotion or DMS cutover plan, prevent_destroy temporarily lifted through a reviewed break-glass PR, maintenance window, rollback rehearsed. Terraform executes steps within that plan; it doesn't get to improvise a database replacement because a plan said so.

Systemic fixes after the near-miss:

• prevent_destroy on every stateful resource (policy-enforced, not convention)
• Policy gate: plans containing replacement of tagged-critical resources fail CI without a migration-approved label
• Deletion protection at the provider level too (deletion_protection = true on RDS/DynamoDB) — defense in depth against non-Terraform deletion paths as well
• Plan rendering in PRs groups replacements at the top — reviewers can't miss what matters

What they're testing: do you read plans like a surgeon reads scans, do you know forces replacement is findable per-attribute, and do you have the organizational reflex — stateful replacement is a ceremony, never a side effect.

The layers, cheapest first:

1. Static (every PR, free): fmt, validate, tflint (catches real bugs: nonexistent instance types, wrong attribute names), security scanners (checkov/trivy) on HCL. Catches typos and policy violations, proves nothing about behavior
2. Plan-based tests: run plan -json against fixture configs and assert on the plan: 'exactly 3 subnets', 'no public ingress', 'all resources tagged'. OPA/Conftest on plan JSON, or terraform test (1.6+) with command = plan — fast, no infra, catches logic errors in conditionals/loops/variable plumbing. This is the highest-ROI layer for module logic
3. terraform test with real applies: HCL-native test files (.tftest.hcl) that apply fixtures, assert on outputs/resources, destroy after — the modern replacement for much of what Terratest did, no Go required
4. Terratest (Go): apply real infra, then probe behavior from outside — HTTP against the ALB, actually connect to the DB, verify the IAM policy denies. Strongest signal, highest cost: real resources, real money, real time (10-40 min), real cleanup discipline (orphan sweepers for failed runs)
5. Example-based smoke: every module ships examples/; CI applies+destroys them nightly against a sandbox account — doubles as living documentation and catches provider-upgrade breakage

What's worth it (the honest allocation):

• Shared modules (the 40-team kind): layers 1-3 on every PR, 4-5 nightly — module bugs multiply by consumer count; this is where testing money pays
• Leaf/app stacks: layers 1-2 only — the plan review IS the test; applying test copies of every app stack costs more than it catches
• The most valuable single test most teams skip: upgrade tests — plan the new module version against state built by the old version; zero-diff (or intended-diff) proves refactors don't replace infrastructure. Catches the count→for_each, renamed-resource class

One-liner: 'test modules like libraries and stacks like configs — and remember the plan is itself a test result someone must actually read.'

Mechanics: drift = reality diverging from state/config. terraform plan refreshes (reads live attributes into state) and diffs — drift appears as unexpected changes in a plan you didn't cause. plan -refresh-only isolates pure drift (what changed outside Terraform) from config changes.

What actually causes it at scale (know the taxonomy — the fixes differ):

1. Emergency manual changes — the 3am SG rule; legitimate, must be reconciled, not just reverted
2. Dual ownership — autoscalers, controllers, or another tool writing attributes Terraform also manages (the ignore_changes cases you haven't declared yet)
3. Cloud-side mutation — AWS retagging, default changes, service-linked modifications
4. Shadow infrastructure — resources created manually that Terraform doesn't know exist at all: invisible to plan (state has nothing to compare) — needs inventory-vs-state comparison (Config aggregator / cloud inventory diffed against state resources), the drift class everyone forgets

The program for 200 stacks:

1. Scheduled detection: nightly plan -refresh-only -json across all stacks (cheap, read-only role); parse results into a drift dashboard — stack, resource, attribute, first-seen. Page nobody; ticket per new drift with the owning team
2. Triage classes → different actions: revert (unauthorized/accidental — apply restores config), adopt (the manual fix was right — PR codifies it; drift count drops by making code match reality), or disown (another system legitimately owns the attribute — explicit ignore_changes + comment)
3. Reduce the inflow: break-glass process that requires a follow-up PR (the 3am change gets a ticket auto-created); IAM so humans can't casually write to Terraform-managed resources in prod (read-heavy human roles; writes via pipeline); tagging standard (managed-by: terraform) making shadow infra queryable
4. Metrics that make it a program, not a hobby: drift MTTR, drift inflow rate per team, % stacks clean, shadow-resource count — reviewed monthly; rising inflow means a process problem (usually break-glass without follow-through), not a tooling problem

The subtle senior point: zero tolerated drift is the goal but auto-revert (apply on cron) is dangerous — reverting a legitimate emergency change mid-incident makes the incident worse. Detection is automated; reconciliation stays a human decision with an SLA.

One-liner: 'drift is unreconciled truth — the program's job is making reality and code converge deliberately, in whichever direction is correct, within a defined SLA.'

The mechanics:

terraform {
  required_version = ">= 1.7.0, < 2.0.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.40" }  # >=5.40, <6.0
  }
}

• Constraints declare acceptable ranges; .terraform.lock.hcl records the exact resolved version + checksums. Commit the lock file — it's the difference between 'CI resolved 5.44 today, 5.45 tomorrow' and reproducible runs. terraform init -upgrade is the only thing that moves it (deliberately, in a PR)
• Pin discipline: ~> pessimistic constraints on minors; never unbounded (>= 5.0) in prod — provider minors ship behavior changes and occasional regressions; you want them arriving as reviewed PRs (renovate on the lock file), not as surprises in Tuesday's apply

Why provider majors are genuinely dangerous: a provider upgrade can change how existing resources are interpreted — removed/renamed arguments (config errors: annoying but visible), changed defaults and resource schema migrations (the dangerous kind: plans suddenly showing in-place updates or replacements on infrastructure you didn't touch). AWS provider majors (3→4's S3 refactor being the infamous case) have forced resource rewrites.

The 200-stack upgrade program:

1. Read the upgrade guide once, centrally — platform team distills it: which resource types we use are affected, what config changes are mechanical vs judgment
2. Codemod the mechanical part: scripted config rewrites where possible, shipped as bulk PRs per team
3. The real gate — plan-diff across the fleet: CI job runs plan with old vs new provider for every stack (read-only), diffing the results. Buckets: zero-diff (mechanically upgradeable — the majority), in-place diffs (review), replacement diffs (stop, investigate per stack)
4. Waves: sandbox accounts → dev stacks → prod by blast-radius order; lock file bumps merge per-stack so rollback = revert the lock PR
5. Modules first: shared modules must support the new major (and declare required_providers ranges honestly) before consumers move

The trap worth naming: provider constraints in modules compose — a module demanding ~> 4.0 blocks every consumer from 5.x until the module updates; modules should declare minimum versions they need (>= 4.9), not ceilings, leaving the ceiling to root configs.

One-liner: 'the lock file makes runs reproducible; the plan-diff across the fleet makes upgrades boring — and provider majors are schema migrations wearing a version number.'

Situation: a botched manual state edit (someone hand-edited JSON to 'fix' a stuck resource) pushed corrupt state; S3 versioning has the pre-edit copy, but three applies happened since — the versioned copy is stale. Prod infra is running fine; Terraform's knowledge of it is wrecked.

Task: restore accurate state with zero infrastructure changes — the infra is the truth; state must be rebuilt to match it.

Action:

1. Freeze: lock the stack (disable the pipeline, hold the state lock) — an apply against bad state is how a bookkeeping incident becomes an infrastructure incident. Communicate: this stack is read-only until further notice
2. Forensics before repair: pull every S3 version of the state (aws s3api list-object-versions) + the corrupt current; diff them. Reconstruct what the three post-snapshot applies changed (CI logs have the plans — this is why plan artifacts are retained). Now you know the delta between the stale-good copy and reality
3. Restore the stale-good version as the base — it's structurally valid and mostly right
4. Reconcile the delta: for each resource the recent applies touched: terraform plan and read carefully —
   • Resource exists in reality, missing/stale in state → import blocks (or state rm + import if half-present)
   • Attribute drift only → plan -refresh-only + apply the refresh (state absorbs reality, no infra change)
   • Resource in state but destroyed in reality → state rm
5. Prove done: terraform plan = zero diff. That's the definition of recovered. Peer-review the final plan output, then unlock
6. The uncomfortable case — no usable backup at all: rebuild state from scratch via bulk import blocks with -generate-config-out assist, resource-by-resource against a cloud inventory. For a big stack this is days — which is the argument for everything in step 7

Result (the real incident this mirrors): ~4 hours to zero-diff for a 150-resource stack; no infrastructure touched.

Prevent (say these unprompted): S3 versioning + replication on the state bucket, object-lock/MFA-delete against deletion, state edits by hand banned — state subcommands only, and even those through a reviewed break-glass; pipeline retains plan/apply logs (your reconstruction record); periodic terraform state pull snapshots to a second location if the stack is critical enough.

What they're testing: the reflex order — freeze, reconstruct truth, rebuild bookkeeping, prove zero-diff — and knowing that state is rebuildable from reality if you stay calm and the infra itself is intact.

The tools:

variable "ingress_rules" {
  type = list(object({
    port        = number
    cidr_blocks = list(string)
    description = optional(string, "managed")
  }))
}

resource "aws_security_group" "app" {
  dynamic "ingress" {
    for_each = var.ingress_rules
    content {
      from_port   = ingress.value.port
      to_port     = ingress.value.port
      cidr_blocks = ingress.value.cidr_blocks
      description = ingress.value.description
    }
  }
}

• Rich types (object, optional with defaults, validation blocks) — make module inputs self-documenting and fail-fast; use freely
• dynamic blocks — generate repeated nested blocks from collections; the legitimate use is exactly the SG-rules case above: data-driven repetition of homogeneous blocks
• Functions + for-expressions — for, merge, try, coalesce, flatten, zipmap: transforming inputs into resource-shaped data

Where the line is (my review heuristics):

1. Nested dynamic blocks = stop. A dynamic inside a dynamic generating conditional content is write-only code; the reader can no longer picture the rendered resource. Restructure the input data instead — do the transformation in locals with named intermediate steps, keep the resource block dumb
2. If you need comments to explain a for expression, decompose it — chains of flatten(for ... [for ...]) become three named locals; each local name documents a step
3. Conditionals selecting shape (not values) are a smell — a module that renders fundamentally different resource sets based on flags is two modules sharing a variables file; split it
4. try()/can() cascades hide contract violations — prefer optional() types with defaults + validation blocks so bad input fails with a message, not a mystery null downstream
5. The render test: can a reviewer predict the plan from reading the module? HCL cleverness that breaks plan-predictability costs more than the duplication it saved

The deeper principle: HCL is deliberately not a programming language — when you're fighting for expressiveness (recursion, real polymorphism), the answer isn't cleverer HCL, it's generating tfvars/JSON from a real language upstream, or reconsidering the module boundary.

One-liner: 'data transformation in locals, dumb resource blocks, types that fail fast — optimize for the reviewer predicting the plan, not the author saving lines.'

The problems it solves (born from real pain at scale):

1. Backend/provider DRY — vanilla TF can't interpolate in backend blocks; 200 stacks = 200 hand-maintained backend configs. Terragrunt generates them from one convention (path_relative_to_include() → state key)
2. Environment hierarchy — terragrunt.hcl files inherit/merge through the directory tree (org → account → region → stack), so per-env config is genuinely just deltas
3. Stack orchestration — dependency blocks wire outputs between stacks (with mocks for plan), run-all walks the dependency DAG applying in order — multi-stack coordination vanilla TF simply doesn't have
4. Hooks, retries on transient errors, bulk operations across the estate

What it costs:

• Another layer: every engineer now debugs two tools' interactions; error messages get indirect; onboarding includes 'why does our Terraform look different from the docs'
• Convention lock-in: the directory hierarchy IS your architecture — restructuring it is a migration; strong opinions that fit or fight your org
• run-all plan/apply semantics have sharp edges (mocked dependencies in plans can mask real issues; blast radius of a tree-wide apply)

Has vanilla TF caught up? Partially — and it changes the calculus:

• Backend partial config + -backend-config flags, terraform test, moved/import blocks, optional() types, stacks (TFC feature) close some gaps
• Still missing in vanilla: cross-stack dependency orchestration, true config inheritance, run-all equivalents. Those remain Terragrunt's moat (or your CI pipeline grows bespoke scripts that are Terragrunt-but-worse)

My decision rule: <20 stacks with a good pipeline → vanilla + conventions; the pipeline handles ordering (it's usually simple). 50+ stacks across many accounts/regions with real inter-stack dependencies → Terragrunt (or a TACOS platform — Spacelift/env0 — solving the same layer with a UI and RBAC). Never adopt it for one feature — adopt it if you'd otherwise rebuild it in CI scripts.

One-liner: 'Terragrunt is the orchestration layer Terraform never shipped — pay its complexity tax exactly when your stack count and dependency graph would otherwise force you to write it yourself, badly.'

The provider mechanics:

provider "aws" {                    # default: the target account
  region = "us-east-1"
  assume_role { role_arn = "arn:aws:iam::${var.account_id}:role/terraform-deploy" }
}

provider "aws" {                    # alias: cross-account resources in one stack
  alias  = "network"
  assume_role { role_arn = "arn:aws:iam::${var.network_account}:role/terraform-network-read" }
}

resource "aws_route53_record" "app" {
  provider = aws.network              # explicit per-resource targeting
  ...
}

The identity architecture:

1. One deploy role per account (terraform-deploy), trust-policied to the CI runner's OIDC identity (GitHub/GitLab OIDC federation — no static keys anywhere), permission-scoped per stack tier where feasible (network role ≠ app role)
2. CI authenticates once (OIDC → org automation account), then assume-role fans out to target accounts — CloudTrail shows sourceIdentity end-to-end
3. Cross-account references prefer data sources with explicit aliased read-roles over god-roles that can touch everything

Module design rule: modules receive providers, never configure them (configuration_aliases in required_providers for multi-provider modules) — a module that hardcodes assume-role is unusable outside its birth account.

The bootstrapping chicken-and-egg (Terraform needs a backend + roles that Terraform is supposed to create):

1. Genesis, once per org: a tiny bootstrap stack run by a human with initial credentials creates: the state bucket (+ locking, versioning, KMS), the OIDC provider, and the automation-account base roles. Its own state starts local, then migrates into the bucket it just created (init -migrate-state) — the one legitimate local-state moment
2. Account vending closes the loop: new accounts are created by an accounts stack (Organizations/Control Tower via Terraform), whose baseline module stamps the terraform-deploy role + trust into every new account at birth — so no account ever needs manual credential setup again
3. Break-glass documented: if automation identity is ever wrecked, the recovery path is the org management account's root ceremony — tested on paper, tightly guarded

What they're testing: OIDC-not-keys reflex, providers-as-injected-dependencies module hygiene, and whether you've actually solved genesis (everyone's first multi-account setup has a hand-made role nobody remembers creating — until DR day).

Situation: ~600 hand-built resources across VPC, IAM, RDS, ECS, a decade of tags conventions, no documentation, original builders gone. Mandate: everything under IaC.

Task: bring it under Terraform management without breaking anything — and push back on 'everything', because 100% import is rarely the right spend.

Action:

1. Inventory first: cloud-side enumeration (Config aggregator, resource explorer, per-service listing scripts) → spreadsheet of resource, type, owner-guess, criticality, change-frequency. This drives sequencing and scoping, not just import commands
2. Scope ruthlessly — the 'leave out' list: ephemeral/auto-created resources (ASG-spawned instances, service-linked roles), things scheduled for decommission, and resources owned by other tools (k8s-controller-created LBs — importing those creates dual-ownership fights). Also defer: stable, never-changed, low-risk singletons — value of importing them is low; risk of touching them isn't
3. Design target structure BEFORE importing: the state layout (network / security / data / per-service stacks) and modules you want — importing 600 resources into one flat main.tf recreates the mess in HCL. Resources import into their destination structure
4. Import in dependency order, stack by stack: network → security/IAM → data → services. Per stack: import blocks + plan -generate-config-out for a first-draft config → refactor draft to real module calls → iterate plan until zero-diff → PR with the zero-diff plan as evidence → next stack. Zero-diff is the non-negotiable gate: any actual change during import phase is a separate, deliberate PR after
5. The gnarly bits:
   • IAM: generated configs are verbose policy JSON — normalize into your policy modules carefully; a subtle diff here is a security change
   • Anything immutable-adjacent (RDS params, launch configs): watch for attributes the provider models differently than the console created — source of phantom diffs needing ignore_changes decisions with comments
   • Tags: adopt-then-standardize — importing + retagging in one motion doubles the review surface
6. Ratchet, don't boil the ocean: after each stack lands, enable drift detection + IAM write-restriction on that slice (humans lose casual write access as Terraform gains ownership) — locking in progress so the environment can't drift back

Result pattern from doing this: ~450 of 600 imported over a quarter (the rest deliberately excluded/deferred with a written rationale), zero incidents during import (the zero-diff gate works), and — the real win — the next change to that environment was a reviewed PR instead of console archaeology.

What they're testing: sequencing sense, the zero-diff discipline, and the judgment to scope — 'import everything' answered with 'import what changes, exclude what's owned elsewhere, defer what's inert' is the senior move.

The layers, innermost out — each catches what earlier layers can't:

1. In-language (module authors): validation blocks on variables, precondition/postcondition on resources/outputs (lifecycle { precondition { condition = data.aws_subnet.this.availability_zone == var.az } }) — contract enforcement inside the abstraction, fails at plan with a message the caller understands. Belongs here: input sanity, cross-resource invariants the module owns
2. Org policy on plan JSON (the main gate): OPA/Conftest (or Sentinel on TFC) evaluating terraform show -json plan.bin in CI:
   • Deny rules: no 0.0.0.0/0 ingress except tagged exceptions, no unencrypted storage, no public S3, mandatory tag schema, no IAM wildcards, replacement of protected resource types requires break-glass label
   • Plan-aware rules (the reason it's plan-time, not static): 'this apply destroys a subnet', 'this changes a prod IAM policy' — behavioral gates static HCL scanning can't see
   • Policies versioned in their own repo, tested (OPA has a unit test framework — policies without tests rot), with severity tiers: hard-fail vs warn-with-override (labeled approval)
3. Static scanning (checkov/tfsec/trivy): fast pattern checks on HCL pre-plan — cheap early signal, catches the obvious in seconds before the plan spends minutes. Overlaps with layer 2; keep it as fast-feedback, let plan-JSON policy be authoritative
4. Cloud-side guardrails (the backstop): SCPs (deny region/root/CloudTrail-tamper), permission boundaries on the deploy roles themselves, AWS Config rules — because Terraform isn't the only path to the cloud; policy that lives only in TF CI is bypassed by anyone with a console. The deploy role's IAM is itself policy: a role that can't delete databases makes the OPA rule defense-in-depth, not the only wall

Design decisions that matter:

• Exceptions as first-class: a policy-exception label + justification field + expiry, reported weekly — a policy system without a paved exception path gets bypassed culturally instead
• Error messages teach: 'S3 bucket lacks encryption — add server_side_encryption_configuration or use corp module s3-standard' beats 'DENIED by rule 47'
• Measure friction: overrides-per-week and time-lost-to-false-positives are policy-quality metrics; a rising override rate means rules are wrong, not engineers

One-liner: 'validate contracts in the module, adjudicate behavior on the plan, backstop in the cloud itself — and make the exception path as well-engineered as the denial path.'

The problem shape: replacement-forcing changes (new AMI, instance family, subnet re-architecture) on serving infrastructure — Terraform's default destroy-then-create is an outage; even create_before_destroy alone doesn't sequence traffic.

The patterns, by mechanism:

1. Let the ASG do the rolling (preferred for fleets): Terraform updates the launch template; the ASG's instance_refresh block (min_healthy_percentage, warmup) rolls instances gradually behind the LB. Terraform's apply is instant and safe — it changed a template; the refresh choreography is delegated to AWS with health-gated progress. replace_triggered_by ties refresh to template changes where needed
2. create_before_destroy + health gates (for singletons with identity): CBD on the resource, name_prefix to dodge collisions, and — critically — health verification between create and destroy: provider-level waits (wait_for_capacity_timeout, target group health), or a two-apply choreography (apply new → verify externally → apply removal PR). One-apply atomicity is a lie for stateful cutover; two reviewed applies with verification between is honest
3. Blue-green at the Terraform level (for the big stuff): new stack (green) as added resources — new ASG/target group/even VPC — traffic shifted by weighted target groups or DNS weights (also Terraform-managed, changed gradually across applies), then blue removed in a final PR. State reflects both stacks during transition; module design must allow N parallel instances (this is where for_each over {blue, green} earns its keep)
4. Data-tier replacements: never a Terraform-native replace — replication/promotion or blue-green features of the service (RDS blue-green, DMS), with Terraform managing endpoints and the final state; the migration itself is a runbook Terraform serves, not drives

Cross-cutting requirements:

• Quota headroom — every CBD/blue-green pattern runs 2x briefly; discover the quota wall in staging, not mid-cutover
• Rollback symmetric: at every intermediate state, a revert PR must cleanly restore — test the revert path for at least the first wave
• Connection draining/deregistration delays tuned so 'instance removed from TG' means 'zero in-flight requests', or the 'zero-downtime' replacement drops the long tail

What they're testing: do you know Terraform is the orchestrator of intent, not the traffic manager — the senior answer delegates rolling mechanics to ASG/LB primitives, keeps applies small and reversible, and never lets a single plan both create the new world and destroy the old one for anything stateful.

The two axes people conflate: where stacks live (env/infra configs) and where modules live (shared abstractions) — they can differ, and the best answers usually do.

Stacks in a monorepo:

• Pros: atomic cross-stack changes (rename a module input + update all callers in one PR), one CI setup, trivial code search, dependency graph visible in one tree
• Cons: CI must do path-based change detection or every PR plans 400 stacks; blast radius of repo-wide mistakes (a bad shared file); merge-queue contention at high team counts; permissions are per-path hacks (CODEOWNERS as ACL)

Stacks per team/domain (polyrepo):

• Pros: real ownership boundaries (repo perms = team perms), independent CI cadence, smaller cognitive surface per repo
• Cons: cross-cutting changes become N-repo campaigns (renovate/scripted PRs — build this muscle or die by it), drift in CI/pipeline versions across repos, discovery ('where is the VPC defined?') needs a catalog

Modules — the sharper question: modules in the stack monorepo referenced by relative path means every module change is instantly live for every consumer — no versioning, no gradual rollout: one bad module edit plans changes across the whole estate. So:

• Modules get versioned distribution regardless of repo layout: git tags per module (monorepo tags like modules/vpc/v2.3.0 work fine), a registry (TFC/Spacelift/artifact-based), and consumers pin versions. The unit of module release is a tag, never a branch tip
• Module repo layout: either a modules monorepo (shared CI, consistent testing, one renovate target — my default) or repo-per-module for very large orgs (independent semver clarity, heavier operational overhead)

My recommendation shape for ~40 teams: modules monorepo with per-module tags + registry; stacks in a small number of domain repos (platform-infra, data-infra, per-large-team repos) rather than one mega-repo or 40 micro-repos — coarse enough for atomic domain changes, fine enough for real ownership. CI: path-filtered plans, merge queue, fleet-wide module bump automation (renovate) with plan-diff evidence per consumer.

The invariant that matters more than layout: consumers pin module versions, upgrades arrive as reviewable PRs with plans, and someone owns the fleet bump muscle — get those three right and either repo topology works; get them wrong and both fail identically.

One-liner: 'the repo debate is mostly a proxy for versioning discipline — version the modules, path-filter the CI, and draw repo lines where ownership actually lives.'

The layering that works:

Terraform owns:   cluster + cloud plumbing
  EKS cluster, node groups/Karpenter infra, OIDC provider,
  IRSA roles, VPC/subnets/SGs, ECR, cluster add-on *infrastructure*
─────────── handoff line ───────────
GitOps/Helm owns: everything with a pod in it
  platform add-ons (via Argo app-of-apps), workloads,
  namespaces-as-tenancy, quotas, NetworkPolicies

Why not manage apps with the kubernetes/helm providers? The trap has layers:

1. Two reconciliation models fight: Terraform converges when applied; Kubernetes controllers converge continuously. Fields controllers mutate (replicas via HPA, injected sidecars, defaulted values) show as perpetual drift → teams sprinkle ignore_changes until Terraform manages nothing but YAML-shaped hope
2. CRD chicken-and-egg: provider schemas resolve at plan time — planning a CR whose CRD doesn't exist yet fails; ordering CRD-install → CR-create inside one graph is fragile (the classic 'works on second apply' stack)
3. Deploy cadence mismatch: app deploys are many-per-day, need progressive delivery, per-service rollback, developer self-service — infra pipelines with state locks and plan reviews are the wrong shape; you end up rebuilding ArgoCD badly inside CI
4. State bloat + blast radius: hundreds of k8s objects in TF state means app changes contend on infra state locks, and an infra refactor can plan app deletions

Where the k8s/helm providers ARE right: the bootstrap seam — Terraform installs exactly one thing into the cluster: ArgoCD (helm_release) + its root Application (kubernetes_manifest), then never touches in-cluster state again. Also fine: aws-auth/access entries, and genuinely infra-coupled singletons (storage classes tied to created KMS keys) — items that change at infra cadence with infra ownership.

The seam contract: Terraform exports what GitOps needs — IRSA role ARNs, subnet/SG IDs, cluster endpoints — via outputs written to SSM parameters or a config repo commit; charts consume them as values. Clean unidirectional flow: cloud facts → cluster config, never circular.

One-liner: 'Terraform builds the stage, GitOps runs the play — the providers exist for the handoff moment, and every workload managed by Terraform is a future drift ticket with your name on it.'

The thesis: cost decisions are made at design time (instance class, storage tier, NAT topology, retention) but discovered at bill time — 4-8 weeks later. Terraform is where infrastructure is decided, so it's where cost belongs surfaced.

The workflow pieces:

1. Cost-diff on every PR (Infracost or equivalent): parses the plan, prices the delta, posts 'this PR: +$1,240/mo (NAT GW ×3, gp3 2TB...)' as a comment. The behavioral effect is the point: engineers see prices while choosing, reviewers see cost as a review dimension. Gates tiered: informational < $200/mo delta; team-lead approval above; FinOps sign-off for step-changes
2. Policy-level cost rules (OPA on plan): deny known money-fires — unattached EIPs, gp2 (gp3 exists), oversized default instance types in modules, provisioned-IOPS without justification tag, multi-AZ NAT when the env is dev. Modules encode the frugal default; policy catches the bypass
3. Tagging as the attribution substrate — enforced, not requested: default_tags at the provider level (blankets everything the provider supports) + policy denying resources missing team/cost-center/env — because every downstream FinOps capability (showback, anomaly attribution, rightsizing targeting) dies without tag coverage. Provider default_tags + module-level merge pattern gets coverage to ~98% without per-resource toil
4. Lifecycle hygiene in code: retention/expiry as module defaults — S3 lifecycle rules, log retention (retention_in_days — CloudWatch's never-expire default is a slow leak), snapshot cleanup, count = var.env == "dev" ? ... scheduling hooks for dev shutdown. Cheap-by-default modules mean cost optimization happens by adoption rather than campaign
5. Close the loop with reality: monthly job reconciles tagged spend vs plan-time estimates per stack — big divergences (data transfer! — invisible to plan-time pricing) get investigated; the estimate model improves; teams see est-vs-actual on their dashboards

Honest limits to state: plan-time pricing can't see usage-driven costs (data transfer, request pricing, autoscaled capacity) — it prices the shape, not the traffic; that's why step 5 exists, and why cost-per-unit telemetry (from the FinOps side) complements rather than duplicates this.

One-liner: 'put the price tag on the PR, make the frugal choice the module default, enforce the tags that make attribution possible — and reconcile estimates against the bill so the system learns.'

The goal: every service PR gets an isolated, production-shaped environment — app + its infra dependencies — created on open, destroyed on merge/close.

The architecture:

1. A purpose-built preview module: the service's stack parameterized by env_id (PR number/branch slug) — every name, DNS record, and tag derives from it. This module is designed for disposability: small instance classes, single-AZ, short retention, spot where possible — production-shaped ≠ production-sized
2. State per instance: backend key templated per env (previews/pr-1234/tfstate) — CI passes -backend-config; workspaces also fit here legitimately (same code/creds/shape, disposable) — pick one and standardize
3. Lifecycle automation: PR opened/updated → plan+apply with env_id; PR closed → destroy + state cleanup. The reaper is mandatory: a scheduled job that lists preview states, cross-references open PRs, and destroys orphans (webhook-missed closes, failed destroys) with a hard TTL (72h) — without it, preview environments are a slow-motion budget incident
4. Data strategy (the first hard part): real prod data is a compliance violation; empty schemas hide bugs. Answers: seeded synthetic fixtures (versioned with the app), sanitized subset snapshots refreshed nightly (masking pipeline), or — cheapest — a shared 'preview data tier' where ephemerality applies to compute/config only and data is a stable, isolated-per-env-schema service. Choose per dependency; document which fidelity each preview actually gives
5. Shared vs per-env dependencies (the cost/speed hard part): a full VPC+RDS+MSK per PR = 25-minute spin-up and real money. Split: static substrate (VPC, cluster, shared postgres instance) provisioned once by the platform; previews create only the fast, cheap layer (namespace/schema/queues/DNS/app deploy) — spin-up drops to 2-3 min. The preview module takes substrate IDs as data-source inputs
6. Access + routing: wildcard DNS + cert (*.preview.corp.dev), env URL posted to the PR, authn in front (previews leak features — SSO gate them)

The remaining hard parts to name honestly:

• Cost visibility: tag everything with env_id + PR author; weekly cost-per-preview report — the 40 concurrent previews × forgotten reaper failure mode is real
• Third-party dependencies: rate-limited/sandboxed vendors don't have per-PR tenancy — stub or share, and mark the fidelity gap
• Destroy reliability: destroys fail (dangling ENIs, non-empty buckets — force_destroy on preview buckets); the reaper needs retry + escalation, not fire-and-forget
• Quota math: N concurrent previews × per-env resources vs account limits — previews get their own account for blast-radius and quota isolation

One-liner: 'previews are a product: a disposability-first module on a stable substrate, a ruthless reaper, and an honest answer about data fidelity — the infra is the easy half; lifecycle and cost discipline are the feature.'

The mechanics underneath the symptom:

1. Unknown values: at plan time, attributes of resources being created are (known after apply) — anything derived from them is unknown too. Mostly fine... until an unknown feeds something that must be known at plan: count/for_each ('invalid count argument' — the classic), provider configuration blocks, or module expansion. Terraform can't size the graph on values that don't exist yet
2. Data source timing: data sources read during plan — if they query something a resource in the same plan will create, they either fail (doesn't exist yet) or return stale results; with depends_on, they defer to apply time and their results become unknown values (feeding problem #1)
3. The 'second apply' pattern: first apply creates the prerequisite, second apply's plan can now resolve the data source/count — the config only converges through repeated application. It works, and it's wrong

Why it's an architecture smell, not a quirk: a config requiring N applies has an implicit, undocumented ordering — CI can't know when it's converged, plans reviewed on PR #1 don't show what apply #2 will do (review integrity broken), and fresh-environment builds (DR!) fail in ways the mature environment never shows.

The fixes, by root cause:

• count/for_each on computed values → restructure so cardinality is config-known: iterate over the input variable (for_each = var.azs) not the resource attribute (aws_subnet.private[*].id); the inputs that determine how many must be literals/variables, even if the contents are computed
• Data source reading your own resource → reference the resource directly. Data sources are for other stacks'/hands' infrastructure; reading what you manage is self-lookup indirection with a timing bug built in
• Cross-stack timing → explicit layering: if stack B needs stack A's outputs, that's pipeline ordering (A applies before B plans), not a data source retrying inside B. The dependency belongs in the orchestration layer where it's visible
• Provider-config-needs-created-infra (the EKS case: kubernetes provider needing the cluster endpoint being created) → split stacks: cluster stack, then in-cluster stack. The two-provider-phase problem is the canonical argument for the layering, and fighting it inside one stack produces the flakiest configs in the ecosystem

Legitimate escape hatches (rare, commented): -target for genuine one-time bootstrap sequences (documented as ceremony, never routine), depends_on on data sources when reading genuinely external eventually-consistent things.

One-liner: 'a plan is a promise about one apply — configs that need two are hiding a layer boundary; find it and make it explicit in the pipeline, not implicit in the retry.'

Situation: one monolithic stack — VPC through IAM through 40 services' infra — plan at 25 min (mostly refresh: thousands of API reads), lock contention meaning engineers queue for applies, and every PR's blast radius is theoretically everything.

Task: decompose to per-domain states with zero infrastructure changes and no deploy freeze — the plane stays flying.

Action:

1. Triage relief first (week 1, no surgery): -refresh=false for local iteration + scheduled refresh; parallelism tuning; split CI to plan only on relevant path changes. Buys tolerability while the real fix proceeds. Resist -target culture — targeted applies as routine practice is how state and reality diverge
2. Draw the target map: network / security / data / per-service — the blast-radius principles (ownership, cadence, privilege). Publish it; every subsequent step moves toward this map and nothing moves twice
3. Extract via state mv → new backends, domain by domain, leaf-first: start with the least-referenced domain (some service's infra), not the network everything references:
   • Copy the relevant HCL into the new stack directory (new backend key)
   • terraform state pull from the mono-state; targeted state mv of the domain's resources into a pushed copy for the new backend (or state rm from old + import blocks in new — mv is faster, import is cleaner audit; pick per domain)
   • Zero-diff plans on BOTH stacks = extraction proven; PR includes both plan outputs as evidence
   • Delete the HCL from the monolith (already state-rm'd/moved — plan shows no changes)
4. Rewire references as you go: consumers that read extracted resources switch from direct reference to data sources against real attributes (tags/names) — deliberately not terraform_remote_state chains, which re-couple the layout you're decoupling and grant state-read (= secrets-read) too broadly
5. The hub extractions last (network, IAM): by now they're referenced only via data sources; moving them is mechanical repetition of a rehearsed dance. The monolith ends as an empty shell — retire the state with ceremony (final backup, backend key archived)
6. Ratchet as you go: each extracted stack immediately gets: its own CI pipeline, scoped deploy role, drift detection, and CODEOWNERS — the decomposition delivers incrementally; even stopping halfway leaves things strictly better

Result pattern: 14 stacks over ~8 weeks alongside normal delivery; worst plan time 25min → 90s; lock contention gone (per-domain locks); and the near-misses stopped — a services PR physically cannot plan a VPC change anymore.

What they're testing: leaf-first sequencing, the both-sides-zero-diff proof discipline, reference rewiring judgment (data sources over remote-state coupling), and delivering value incrementally instead of proposing the six-month rewrite.

The scenario people reach for these in: 'I only want to apply this part — the rest of the plan is noisy/risky/slow.'

-target — what it actually does: restricts plan/apply to named resources + their dependencies, ignoring the rest of the graph. Sanctioned uses: recovering a wedged state mid-incident, bootstrap ordering ceremonies, surgical drift fixes. Why it's almost always wrong as practice:

1. It applies a state the config never described — the config promises a whole; -target delivers a fragment. Post-target, the next full plan shows the deferred remainder... which now surprises whoever runs it
2. Dependency myopia: the target's dependents aren't updated — you changed the SG but not the things consuming its outputs; the graph's consistency guarantees are exactly what you turned off
3. It's culture-forming: teams that -target routinely stop trusting full plans ('there's always noise'), and untrusted plans stop being read — the review pillar quietly collapses. Noisy plans are a symptom (drift, dual ownership, bad structure); -target treats the symptom while the disease compounds
4. Terraform itself warns on every use — it's an escape hatch with a handle worn smooth

Partial applies via saved plans: apply plan.bin executes exactly the reviewed plan — this is the correct mechanism for 'what runs = what was reviewed', but it's the full plan, just pinned. It solves review-integrity, not scope-reduction. (There's no sanctioned 'apply half the plan' — that's -target wearing a suit.)

Separate backends/stacks — the structural answer: if you keep wanting to apply only part of a state, that part is telling you it's a different stack: different cadence, different risk, different owner. Splitting it makes 'apply only the network change' the normal workflow (you're in the network stack) instead of a graph override. All the -target urges map to missing boundaries:

• 'DB changes are scary in app plans' → data stack
• 'plan is too slow to review' → decompose (see the 6,000-resource playbook)
• 'two teams keep colliding in one state' → ownership boundary

The decision rule I give teams: reaching for -target more than ~once a quarter per stack = a standing architecture ticket, not a habit. Incidents get the escape hatch (documented in the postmortem, followed by a full reconciling plan); everything else gets a boundary.

One-liner: '-target is morphine — right for the emergency, ruinous as a lifestyle; the chronic pain it masks is a missing stack boundary, and the cure is surgery, not dosage.'

The uncomfortable question behind it: everyone plans DR for infrastructure; almost nobody plans DR for the thing that rebuilds infrastructure. If region-loss recovery depends on Terraform, Terraform's own availability is tier-0.

The failure modes and their answers:

1. CI/CD platform down (pipeline outage): applies must be executable from a documented break-glass path — a runner AMI/container with pinned Terraform + provider cache, an operator role assumable by 2-3 named humans (MFA, audited), and the runbook tested quarterly: checkout tag → init against backend → plan → second-human review → apply. RTO target: <1h to emergency-apply capability. Without rehearsal this is fiction — provider downloads alone fail if your artifact proxy is also down (hence the cached runner image)
2. State backend loss (bucket/account compromise): defense layers — bucket versioning + cross-region (ideally cross-account) replication of the state bucket, object-lock against deletion, and scheduled state pull snapshots to an isolated account. Recovery: repoint backend config to the replica (a rehearsed one-line change per stack), verify zero-diff plans. The cross-account copy matters: an account-level compromise that can delete the bucket can delete same-account backups
3. Full region loss (DR invocation): the IaC must be region-parameterized and actually exercised — a DR account/region where core stacks apply cleanly from scratch. This is where 'works on second apply' configs, hardcoded AZs, and un-imported manual fixes surface — the game-day finding list writes itself. Data-layer restore (from backups) is separate; Terraform rebuilds the shape in RTO-minutes only if the config truly describes everything
4. Compromise scenario (malicious insider/supply chain): state and config are attack surfaces — signed commits + protected branches on infra repos, provider/module allowlisting with checksums (the lock file is a security control), plan-stage isolation (plan executes provider code!), and an audit answer for 'what did Terraform identities do this month' (CloudTrail on the deploy roles). Recovery from bad applies: state snapshots + git history make 'rebuild belief to a known-good point' tractable — provided applies were serialized and logged
5. Knowledge/bus-factor loss: the boring one that actually happens — stack inventory (owner, purpose, dependencies, last-applied) as maintained metadata, README-per-stack, and the genesis/bootstrap runbook current — because the person who built the state architecture leaving is a slow-motion DR event

The drill that ties it together: yearly 'IaC blackout' game day — pipeline disabled, primary state region 'lost', rebuild capability proven from replicas + break-glass runner, timed. The measured number goes in the DR doc next to the infrastructure RTOs it gates.

One-liner: 'your infrastructure's RTO has Terraform's RTO as a floor — replicate the state, cache the toolchain, rehearse the break-glass, and treat the pipeline as tier-0 infrastructure because during DR, it is.'

Why they're a trap: provisioners (remote-exec, local-exec) run imperative scripts inside a declarative engine — Terraform can't plan them (no diff — just 'will run script'), can't know if they succeeded semantically, can't undo them, and their effects live outside state. A failed provisioner leaves a tainted resource (destroyed and recreated next apply — often the wrong medicine). HashiCorp's own docs call them a last resort.

Replacement table for the common uses:

• Bootstrapping instances (install agent, configure app) → bake AMIs (Packer) + user_data/cloud-init; config management belongs to the image pipeline, not apply time. Instances should be born correct — that's also what makes autoscaling work without Terraform present
• 'Run this after the DB exists' (schema init, seed) → app migration step in the deploy pipeline, or the provider-native option (many services have init parameters); pipeline steps are visible, retryable, and ordered explicitly
• Calling an API Terraform lacks a resource for → check for a provider first (there usually is one now); else the http data source for reads, or a small custom/scaffolding provider — a real resource with real CRUD beats a script pretending
• Local file generation → local_file resource, templatefile() — declarative, diffable
• Triggering external systems on change → EventBridge/webhook driven by the pipeline post-apply, keyed on output diffs — not local-exec curl

Where null_resource/terraform_data is still legitimate: as a change-detection anchor — terraform_data with triggers_replace to force downstream replacement on input changes, or as a dependency junction. That's using it as graph plumbing, not as a shell-script host.

One-liner: 'a provisioner is an admission that something isn't modeled — model it (image, pipeline step, provider) instead of hiding a script where the plan can't see it.'

How the graph is built: Terraform parses every expression reference — aws_instance.api.id inside another resource creates an edge; module.vpc.subnet_ids creates cross-module edges. The DAG orders operations; independent branches run in parallel (default 10 walkers). This is why referencing attributes rather than re-deriving values isn't just style — it's how ordering correctness happens.

Implicit (reference-based) is always preferred because the dependency is load-bearing data: it can't go stale (delete the reference, the edge disappears with it), and it documents itself.

When explicit depends_on is genuinely correct — hidden dependencies with no data flow:

• IAM propagation cases: the role policy must exist before the Lambda/service that uses it works, but nothing in the consumer references the policy attachment — depends_on = [aws_iam_role_policy.x] encodes the invisible ordering
• Resources coupled through out-of-band effects: a VPC endpoint that must exist before instances boot (their user_data hits S3), an eventual-consistency wait modeled on a predecessor
• Module-level depends_on for coarse sequencing when a module's side effects (not outputs) are prerequisites

When it's a smell:

1. depends_on where a reference would do — someone hardcoded a value instead of referencing the attribute, then patched ordering back with depends_on: two bugs pretending to be a fix. Replace the literal with the reference; delete the depends_on
2. depends_on on data sources — forces the read to apply-time, making results unknown at plan (cascading the count/for_each problems); usually means reading something this same config manages (reference it directly) or a missing stack boundary
3. Sprawling module depends_on chains — sequential module walls destroy parallelism and usually re-encode what output references would express precisely

Debugging tool worth naming: terraform graph | dot -Tsvg when ordering surprises you — cycles (from tangled depends_on + references) and missing edges become visible.

One-liner: 'let data flow define the graph; reserve depends_on for dependencies the data genuinely can't see — and treat every one as a comment-worthy anomaly.'

The modern verbs and what each moves:

• terraform apply -replace=aws_instance.api — plan a forced destroy/recreate of a healthy-looking-but-actually-broken resource (corrupted node, instance that missed a critical boot step). Replaces taint and fixes its flaws: taint mutated state immediately (marked at taint time, replaced whenever the next apply happened — possibly someone else's, as a surprise); -replace lives in the plan, visible and reviewed like any change, atomic with its apply
• terraform plan/apply -refresh-only — read reality, update state only, change no infrastructure: the sanctioned way to absorb drift you've decided to accept (someone resized an instance mid-incident; you're codifying it — refresh-only first so state matches reality, then a config PR so code matches state, then zero-diff proves the triangle closed)
• -refresh=false — skip the reality-read for speed during iteration (big states); the trade: you're planning against possibly-stale state — never for the final pre-apply plan
• replace_triggered_by (lifecycle) — declarative replacement coupling: 'replace the instance whenever the launch template changes'. The recurring-relationship version of -replace's one-time surgical strike
• terraform_data with triggers_replace — anchor arbitrary values (a script hash, an AMI date) so their change forces replacement of dependents — change-detection plumbing for inputs Terraform doesn't natively track

The decision grid: broken resource, once → -replace. Reality changed, accept it → refresh-only + codify. Recurring 'when X changes rebuild Y' → replace_triggered_by. External-input-driven rebuilds → terraform_data triggers.

The discipline thread: all of these keep changes inside the plan/review flow — the shared property that made taint's out-of-band state mutation worth deprecating. If your process still says 'taint it', your process predates 1.x and is worth a refresh itself.

One-liner: 'the modern toolkit makes every forced change and every absorbed drift a planned, reviewable event — taint died because it was neither.'

The interface (small on purpose):

module "vpc" {
  source     = "corp/vpc/aws"   # registry, pinned
  version    = "~> 3.2"
  name       = "payments-prod"
  cidr       = "10.20.0.0/16"    # from IPAM, not invented
  az_count   = 3
  flow_logs  = true              # default true; off needs a reason
  nat        = "per_az"          # enum: per_az | single | none
  tags       = local.tags
}

Internals — computed, not configured: subnets derive from cidrsubnet() so layouts are consistent by construction:

locals {
  # /16 in → 3× /20 private, 3× /24 public, 3× /24 data — same shape every VPC
  private = [for i in range(var.az_count) : cidrsubnet(var.cidr, 4, i)]
  public  = [for i in range(var.az_count) : cidrsubnet(var.cidr, 8, i + 48)]
  data    = [for i in range(var.az_count) : cidrsubnet(var.cidr, 8, i + 64)]
}

Callers give one CIDR; the module guarantees every org VPC has identical structure — which is what makes firewall rules, TGW routing, and runbooks portable across environments.

What's deliberately NOT parameterized (the actual design skill):

• Subnet layout/sizes — configurable layouts mean every VPC is a snowflake; the 3-tier shape IS the standard. Teams with genuinely different needs talk to platform (and maybe that's a second module, not a parameter)
• Gateway VPC endpoints (S3/DynamoDB) — always created; they're free and forgetting them is a NAT bill
• DNS attributes, default-SG lockdown, flow-log format — org policy, not preference
• Public subnets are small (/24 — LBs and NAT only) by design; 'bigger public subnets' is a smell to push back on, not a knob

Outputs as the contract: subnet IDs by tier (private_subnet_ids), route table IDs, VPC/endpoint IDs — shaped for downstream modules (EKS, RDS) to consume without knowing internals.

Versioning reality: VPC modules are the highest-blast-radius modules you'll own — CIDR/subnet changes force replacement of everything inside. Upgrade tests (old-state + new-version = zero-diff) are mandatory, and some changes are honestly 'new VPC + migrate' — say so in the changelog rather than pretending a v4 upgrade is in-place.

One-liner: 'a great VPC module takes one CIDR and returns your org's standard network — the parameters it refuses to have are worth more than the ones it offers.'

First truth: review the PLAN, not just the HCL — the diff shows intent; the plan shows consequence. No plan output attached = not reviewable; that's pipeline table stakes.

The plan-side checklist (in order):

1. Destroy/replace census: any - or -/+? On what? Stateful resources (DB, volumes, buckets) in a replace = full stop, migration-plan conversation. In-place ~ on IAM/SG/network = read the specific attributes (a one-line cidr change can be an exposure)
2. Count sanity: '3 to add' expected from this diff, or is the module fan-out surprising? A 40-resource plan from a one-line change means a module default shifted — find out which
3. Unknown-value cascades: (known after apply) on things that feed decisions — will this actually converge in one apply?
4. The absence check (juniors always miss this): what should be in this plan and isn't? Added a subnet but no route table association; new SG with no rules attached; the resource created but never referenced — half-wired infrastructure passes every syntax check

The HCL-side checklist:

5. Hardcoded values that should be references or variables — literal AMI IDs, account numbers, CIDRs (drift and portability debt); literal secrets (instant block + rotate if it ever hit a commit)
6. Identity churn risk: anything renamed/moved without a moved block; count on collections where for_each belongs (the index-shift replacement bomb waiting for the next PR)
7. Lifecycle honesty: new stateful resource without prevent_destroy/deletion protection; ignore_changes additions — each one gets a 'who's the other owner?' question
8. Module/provider pins: ref=main, unbounded version ranges, new providers not in the allowlist
9. Blast-radius fit: does this change belong in this stack? App PR touching IAM baseline or shared networking = boundary violation regardless of correctness

Context checks seniors add:

10. Cost line (the Infracost comment): delta plausible for the intent?
11. Policy waivers: any exception labels — justified and expiring?
12. Rollback story for the risky ones: 'if this apply goes sideways at 40%, what's the revert path?' — if the answer is 'unclear', the PR needs restructuring into smaller applies, not more approval

The meta-skill: calibrated attention — a tag-only diff gets 30 seconds; anything with -/+, IAM, or network primitives gets the full ceremony. Uniform paranoia reviews nothing well.

One-liner: 'juniors review whether the code is right; seniors review what the plan will do, what's missing, and what the next PR inherits — the plan is the artifact, absence is a finding, and identity churn is the silent killer.'

The maturity ladder (each rung is a real operating model):

1. Curated modules + docs: developers copy an example, fill variables, PR to their stack repo. Cheap, honest, HCL-visible — works to ~20 teams if modules are excellent. The platform team's product is the module registry + golden examples
2. Scaffolded stacks: a generator (Backstage template, CLI) stamps the repo/stack/pipeline with the module pre-wired — developer answers 5 questions, never writes backend config or provider blocks. The generator encodes the paved road; HCL still visible for the 10% who need to extend. This rung + rung 1 covers most orgs
3. Interface-based self-service: developers write a simplified spec — not HCL — and platform machinery renders it:
   • The data-file pattern: service.yaml in the team repo (name, tier, needs: [postgres, queue]) → CI generates tfvars → shared stacks for_each over the parsed files. Terraform-native, reviewable, no new runtime
   • TACOS-mediated (Spacelift/env0/TFC): platform-defined blueprints, developer-facing forms/APIs, RBAC and policy in the platform layer
4. Control-plane self-service: a real API/portal provisions via queue-driven automation that runs Terraform underneath (or replaces it — see below); developers never see plans. This is an internal product with SLOs, versioning, support rotation

The design invariants at every rung: the simplified interface must still produce reviewable, policy-gated changes (self-service ≠ ungoverned); quotas/budgets per team baked into the interface; and an escape hatch to raw HCL that doesn't require re-platforming (the teams who outgrow the abstraction are your most important users).

When it stops being Terraform's job — the honest boundary: Terraform's model is plan-review-apply of declared state — it strains when you need:

• Per-resource-instance lifecycle at high frequency: thousands of short-lived, API-created resources (per-tenant databases created on signup) — that's a runtime concern: queue + idempotent provisioner service, or Kubernetes-resident control planes (Crossplane/ACK: claims reconciled continuously, K8s-native RBAC/quotas)
• Sub-minute provisioning SLOs where plan/lock/apply latency is the product bottleneck
• Continuous reconciliation semantics (drift auto-healed, not detected-and-ticketed)

The pragmatic hybrid most orgs land on: Terraform owns the substrate (accounts, networks, clusters, the Crossplane installation itself); a reconciling control plane owns high-frequency tenant-shaped resources; both are policy-gated by the same OPA corpus so governance doesn't fork.

One-liner: 'self-service is an interface-design problem — keep Terraform underneath while requests are infrastructure-shaped and review-cadenced; the moment they're runtime-shaped (per-tenant, high-frequency, reconciled), give them a control plane and let Terraform build it instead.'