IaC Fundamentals

TL;DR

• Declarative IaC (describe desired state, tool computes diff) is naturally idempotent; imperative scripts (ordered commands) produce different results on re-run depending on starting state.
• State tracking enables accurate planning: the tool must know what it already provisioned to compute correct diffs and avoid unintended deletions or collisions.
• Plan-before-apply is the highest-leverage safety mechanism: reading plan output catches destroy-and-recreate surprises on stateful resources before they happen.
• Modules encode organizational standards once, reviewed once; the alternative is every team independently reimplementing standards and drifting when they change.
• Idempotency (same operation run twice = same result) is the property that makes re-running safe and makes recovery from partial failures straightforward.

Core Concepts

Declarative vs. imperative infrastructure

Declarative: describe desired end state (“this should exist with these properties”); the tool diffs desired against actual and applies only what’s needed.

Imperative: sequence of commands (“run this command, then that one”); the tool executes instructions in order, and re-running produces different results depending on what state it started in.

# Declarative — idempotent, safe to re-run
resource "aws_s3_bucket" "data" {
  bucket = "my-bucket"
  acl    = "private"
}
# Running this 10 times produces the same result

# Imperative script — NOT idempotent, state-dependent
#!/bin/bash
aws s3api create-bucket --bucket my-bucket  # Fails if bucket exists
aws s3api put-bucket-acl --bucket my-bucket --acl private
# Re-running fails because the bucket already exists

The practical consequence: a declarative tool re-run with no changes produces no changes. An imperative script re-run is a risk.

State management: the hidden enabler of safety

State is the mapping between configuration definitions and real infrastructure. The tool tracks what it provisioned (resource IDs, attributes, current values) so it can accurately compute a diff on the next run. Without state:

• Can’t distinguish “resource doesn’t exist yet, create it” from “resource exists, don’t touch it”
• Can’t detect drift (actual state diverging from desired)
• Can’t safely move or rename resources without recreating them
• May delete something it shouldn’t, or try to recreate something that already exists

State loss or corruption is a severe incident class. Teams with good IaC practice store state durably (remote backend with encryption, not local files), with locking to prevent concurrent conflicting applies.

Modules: standardization through reuse

A well-tested, shared VPC module encodes organizational standards (correct CIDR ranges, required tags, baseline security rules) once. Every consuming team gets those standards automatically. Without modules:

• Every team hand-writes the same infrastructure independently
• Standards drift the moment one team forgets a rule or interprets it slightly differently
• When a standard changes, propagating the change everywhere is a coordination nightmare

Plan review and the destroy-and-recreate gotcha

The plan output shows exactly what will change. The most dangerous surprise:

Plan: will destroy and recreate aws_db_instance.prod

A minor property change (maybe just the password) shouldn’t require destroying and recreating a database — that causes data loss. Reading the plan catches this before applying, and usually points to a configuration restructuring or using a resource-specific update mechanism instead of a destroy-and-recreate.

This is the single highest-leverage habit: reading plan output before applying it, with specific attention to any destroy-and-recreate on stateful resources.

Idempotency and recovery

Idempotency (same operation run twice = same result) is the property that makes:

• Re-running safe after a partial failure
• Recovery from human errors straightforward
• Drift correction safe to run repeatedly

A tool that’s genuinely idempotent means “just run it again” is a safe, boring response to a failed or interrupted operation.

IaC Tools Compared

| Tool | Primary use | Declarative | Multi-cloud | State model | Typical org size | |—|—|—|—|—| | Terraform | Cloud infrastructure | Yes | Yes (native) | Explicit state file | Any | | CloudFormation | AWS infrastructure | Yes | AWS only | Managed by AWS | AWS-heavy | | CDK | AWS infrastructure | Yes (→ CloudFormation) | AWS only | Managed by AWS | AWS-heavy, code-heavy | | Ansible | Configuration, provisioning | Partial (modules idempotent) | Yes | Stateless (queries live system) | Any | | Pulumi | Cloud infrastructure | Yes (→ cloud provider) | Yes | Language-dependent | Dev/engineering-focused | | Bicep | Azure infrastructure | Yes | Azure only | Managed by Azure | Azure-heavy |

Design Review Checklist

[ ] State is stored remotely (S3, Terraform Cloud, etc.), not locally
[ ] State is encrypted at rest and in transit
[ ] Concurrent applies are prevented (state locking enabled)
[ ] Plan output is reviewed before every apply, with attention to destroy-and-recreate on stateful resources
[ ] Shared infrastructure patterns are built as modules, not copy-pasted
[ ] All state backend configuration is documented and version controlled (except secrets)
[ ] Idempotency is confirmed: re-running with no changes produces no changes
[ ] Code review process treats IaC changes with same rigor as application code changes

Common Pitfalls

• Pitfall: Plan shows destroy-and-recreate on production database; applied without close review; data lost. Why: Plan output was scrolled past instead of read carefully. Fix: Make plan review a mandatory, enforced step; block applies that show destroy-and-recreate on stateful resources without explicit justification.

• Pitfall: Every team hand-writes its own VPC configuration; each slightly differently missing an organizational security standard. Why: No shared module encoding organizational standards. Fix: Build and maintain shared modules for common patterns (VPCs, security baselines, logging); consume rather than reimplement.

• Pitfall: State file lost or corrupted; takes hours to reconstruct what infrastructure actually exists. Why: State stored locally or in a fragile location; no backup or recovery procedure. Fix: Use a remote backend with encryption, versioning, and tested backup/recovery procedures from day one.

• Pitfall: Imperative provisioning script produces different results on repeated runs, causing “works sometimes” infrastructure. Why: Imperative commands don’t check current state before acting. Fix: Switch to declarative tooling; if stuck with imperative, add idempotence checks (e.g., “does X exist? if not, create it”).

• Pitfall: Applying without reading the plan; unexpected destroy-and-recreate causes outage. Why: Plan output wasn’t reviewed. Fix: Make plan review a mandatory, block-applies-without-review step in CI.

Interview Questions

• A plan shows a database resource will be destroyed and recreated due to a property change. What’s your next step, and why? — Tests whether plan-review discipline is a genuine habit, not a formality.

• Why does state management matter as much as the declarative definitions themselves? — Tests understanding of the diff-computation dependency on accurate state tracking.

• Design the case for building a shared VPC module instead of letting each team write their own VPC configuration. — Tests whether governance and consistency value is understood beyond mere convenience.

• What’s the difference between a tool that’s truly idempotent and one that claims to be? How do you verify? — Tests understanding of re-run safety and verification methods.

• A state file is lost. Walk me through recovery. What could have prevented this? — Tests understanding of backup strategies and state durability requirements.