Log Aggregation

The Pipeline

 logs scattered across pods/hosts/lambda-ephemera are useless
 at 3am. AGGREGATION centralizes them:

 [app writes JSON]──stdout/file──►[collector agent]
                                     │ batch, compress, tag
                                     ▼
                          [transport: kafka-class buffer]
                                     │
                                     ▼
                     [storage/index: loki/elastic/ch]
                                     │
                              [query UI/alerts]

 design points per stage:
 - agent: lightweight (fluentbit/vector-class), backpressure-aware
 - BUFFER between transport and storage: log spikes must not
   lose data OR take down the pipeline (kafka absorbs bursts)
 - indexing strategy: full-text (elastic) vs label-indexed
   (loki) = the cost/flexibility fork of this whole lesson

Retention Tiers

 keep everything hot forever? bankruptcy. tier it:

 HOT (days):    instant search — active incidents live here
 WARM (weeks):  cheaper storage, slower queries — investigations
 COLD (months/year): compressed archives, compliance access,
                re-hydrate on demand only

 retention by CLASS not globally:
   error/security/audit logs: long (compliance mandates!)
   info business events:      medium
   debug/sampled:             short or none

 cost lever ranking (biggest first):
 1. volume reduction at source (structured-logging discipline)
 2. retention tiering  
 3. index strategy (label-only vs full-text)

Operational Realities

Concern	Practice
pipeline down	apps buffer locally; disk-caps prevent node death
missing logs mystery	agent health metrics + drop counters ALERTED
cross-service correlation	trace_id indexed as first-class field
multi-region	regional ingest + global query federation
schema drift	versioned event schemas; lint at CI

 the silent-drop failure is the classic:
 collector overwhelmed → drops "temporarily" → nobody notices
 until an incident needs exactly those logs.
 drop-counters and end-to-end volume dashboards are mandatory,
 not nice-to-have. you must KNOW what you didn't keep.

Interview Framing

“Design logging for a 200-microservice platform” scored shape: staged pipeline drawn with buffering emphasized, retention-tiering by log CLASS, index-strategy tradeoff articulated (loki-vs-elastic class), silent-drop monitoring called out unprompted, source-volume discipline linked. Aggregation questions test whether you’ve operated pipelines — the failure modes are where that shows.