Schema Evolution

Data Outlives Code

Every persisted record, queued message, and cached blob encodes an older schema. Deploy v2 code while v1 data fills the database — evolution isn’t an event; it’s a permanent condition:

 during ANY migration window:
   old writers + new readers coexist
   new writers + old readers coexist
   (rolling deploys make this true even for minutes)
 
 schemas must therefore be read/write across versions
 by design, not by luck

The Compatibility Matrix

Direction	Meaning	Example
Backward	New code reads OLD data	New consumer handles last year’s events
Forward	Old code reads NEW data	Pre-upgrade consumers survive a producer deploy
Full	Both	Rolling deploys with zero coordination

 ADD optional field          backward ✓ forward ✓ (tolerant readers)
 REMOVE field                depends on reader strictness
 RENAME field                ✗ both — it's remove+add
 WIDEN int32→int64           protobuf-safe (varint), JSON-ambiguous
 NARROW type                 ✗ silent truncation risk
 ADD required field          ✗ breaks all existing producers

The Expand–Migrate–Contract Pattern

For database schema changes specifically:

 phase 1 EXPAND:    add new column nullable / new table
                    old code unaffected; dual-write begins
 phase 2 MIGRATE:   backfill historical rows in batches
                    verify counts/checksums
 phase 3 CONTRACT:  switch reads to new column,
                    stop dual-writing, drop old column LATER
                    (after confidence period)

 each phase deploys independently; no big-bang cutover;
 rollback = revert one phase, not the world
 
 the final DROP waits weeks: old replicas, caches, and
 forgotten batch jobs may still read the dead column

Message Schemas

Queued events are the hardest case — consumers upgrade asynchronously, possibly hours behind producers:

 rules that keep event streams safe:
 - producers: additive changes only; never reuse field ids (protobuf)
 - consumers: tolerant readers (ignore unknown fields)
 - registry enforces compatibility mode per topic:
     BACKWARD (default): new schema reads old messages
     FULL:               both directions enforced at publish time
 
 breaking change = NEW topic/version, run both, migrate, retire

Deprecation Observability

 before removing anything, measure actual usage:
 - log reads of legacy columns/fields with caller identity
 - dashboards per deprecated field
 removal ships when traffic → zero, not when the ticket says so
 
 forgotten consumers are the norm; observability finds them

Interview Framing

“Schema changed — what breaks?” tests operational maturity. Scored shape: name the compatibility directions unprompted, apply expand-migrate-contract for stored data, note registry-enforced modes for events, close with deprecation telemetry. The word “dual-write” alone signals real-world migration experience.