Menu

Earn Premium with Referrals

Invite your friends and earn Premium rewards through our referral program.

See how it works and start inviting friends.

Storage Growth Math
HLD

Storage Growth Math

Projecting data accumulation over years — linear vs compounding growth, retention levers, and capacity cliffs.

The Base Projection

 yearly accumulation = daily new bytes × 365

 RideShare trips:   20 GB/day × 365        ≈ 7.3 TB/year
 chat messages:     2B × 500 B = 1 TB/day  ≈ 365 TB/year
 image host:        20 TB/day              ≈ 7.3 PB/year (!)

 the multiplication alone often picks the architecture:
 TB/year → ordinary databases fine; PB/year → object storage + lifecycle tiering from birth

Linear vs Compounding

Growth shape matters more than growth rate:

 LINEAR (stable user base)         COMPOUNDING (user growth g per year)
 year n = year-1 bytes             year n ≈ year-0 × (1+g)ⁿ
 
 flat slope                        g=50%/yr: year-5 ≈ 7.6x year-0
                                   g=100%/yr: year-5 ≈ 32x year-0
 
 planning error mode:              planning error mode:
 none — just keep buying           "we have room for years" →
                                    full in 18 months
 
 real products compound until maturity, then bend linear:
 model BOTH phases, find the knee, plan procurement against it

Retention: The Only Real Lever

Storage problems are solved by deleting or demoting data, never by bigger disks alone:

 retention policy effect on steady state:

 forever:      grows without bound → eventually a migration crisis
 TTL 90 days:  steady state ≈ 90 × daily rate → CONSTANT forever
 tiered:       hot (SSD) shrinks to recent window;
               cold (object storage) absorbs history at 1/10th cost

 example: 20 GB/day trip data
   all-hot forever:    unbounded Postgres growth ✗
   90-day hot window:  ~1.8 TB hot, stable ✓
   older → object store at ~$10/TB-month class pricing (illustrative)

Compliance floors set minimum retention; product need sets everything above the floor. The gap between them is free money.

Multipliers Revisited on Growth

Every projection must carry the storage lesson’s factors before comparing to hardware:

 usable = raw × index-factor(1.2–2) × replication(×3 RF / ×1.4 EC) ÷ max-utilization(0.7)

 RideShare 5-year check:
 raw 36.5 TB → ×1.5 indexes → ×3 replication → ÷0.7 ≈ 235 TB provisioned
 → fits comfortably in one large cluster's budget... 
 → but the IMAGE HOST at PB-scale never had that option — tiering from day one

Capacity Cliffs

Watch for discontinuities, not slopes:

CliffTriggerConsequence
Index memory exceeds RAMworking set outgrows nodelatency regime change
Single-node disk ceilinglargest replica can’t fitforced resharding project
Backup window exceeds daydump time > 24hbackup architecture redesign
Cross-region lag > freshness SLOdata volume vs pipeconsistency strategy shift

Each cliff converts a smooth growth curve into a mandatory engineering quarter — spotting them early is the entire purpose of growth math.

Interview Framing

Strong candidates present growth as three numbers: today’s daily rate, the projected shape (linear/compounding with stated assumption), and steady-state-under-retention. Then they name the first cliff (“at current shape, the index outgrows RAM in ~2 years — archive policy starts now”). That final sentence — a dated cliff with an owner — is what separates estimation from planning.

My Private Notes

Notes are auto-saved locally to this device.