Immutable Infrastructure

The Philosophy

 MUTABLE (pets): servers accumulate state via SSH patches:
   apt upgrades, hand-edited configs, hotfix scripts...
   → configuration drift: every server slightly unique,
     "works on prod-server-3" mysteries, unrepeatable builds

 IMMUTABLE (cattle): servers are NEVER modified after birth:
   change = build NEW image → replace instances → discard old
   
 [v1 image] [v1] [v1] ──deploy──► [v2 image] [v2] [v2]
   old ones destroyed, never patched in place ✓

 the artifact IS the environment: what you tested is
 byte-identical to what runs. drift becomes impossible
 by construction.

The Layered Immutability Stack

 each layer replaced, not mutated:

 □ CONTAINERS: docker/OCI images — the standard unit;
   digests pin exact bytes
 □ MACHINE IMAGES: AMIs/cloud-init baked VMs for node pools
 □ INFRA DEFINITION: terraform/pulumi — infra changes via
   plan+apply producing new resources, not console clicks
 □ CONFIG: injected at launch from versioned sources
   (never hand-edited on running boxes!)

 enforcement matters more than intention:
 ✗ disable/audit SSH access to app servers
 ✗ read-only root filesystems; ephemeral everything
 ✗ console-write policies revoked (terraform-only changes)
 drift detection scanners catch violators anyway.

Gain	Mechanism
test-prod parity	same digest everywhere
instant rollback	redeploy previous digest
security hygiene	fresh patched base images per build
scale-out confidence	new instance ≡ existing instances
disaster recovery	infra rebuildable from repo + registry

The Costs and Edge Cases

 □ IMAGE BUILD TIME in the deploy path (mitigate: layer
   caching, base-image reuse, parallel builds)
 □ REGISTRY/STORAGE growth: prune policies required
 □ STATEFUL components resist immutability: databases can't
   be "replaced" casually — immutability applies to the
   SURROUNDING infrastructure; data keeps its own disciplines
   (backups, migrations)
 □ EMERGENCY HOTFIX temptation: 3am SSH fix feels faster...
   and silently forks reality. answer: fast image pipeline
   (minutes) so the correct path is also the fastest path.
 □ SECRETS: never baked into images! runtime injection only.

Interview Framing

“Our servers drifted; staging matches nothing anymore” scored shape: mutable-vs-immutable diagnosis immediately, layered stack named (images/terraform/config), enforcement mechanisms beyond good intentions (SSH lockdown, drift scanners), stateful-exception honesty, hotfix-pipeline-speed insight. Immutable-infrastructure questions test whether you prevent drift by ARCHITECTURE or by hoping engineers stay disciplined — hope isn’t a mechanism.