Webhooks

The Inversion

REST: consumer polls provider. Webhooks: provider pushes to a consumer-registered URL when events happen.

 registration:
   POST /webhook_endpoints
   { url: "https://acme.com/ride-hooks", events: ["trip.completed"] }

 delivery (later, event occurs):
   POST https://acme.com/ride-hooks        ← PROVIDER calls CONSUMER
   X-Ride-Event: trip.completed
   X-Ride-Signature: t=1690,v1=hmac(...)   ← auth!
   { trip_id, fare, completed_at }

Why push wins for the right cases: polling wastes calls when events are rare (payments, deployments) and lags when they’re urgent. One webhook replaces millions of empty polls.

Delivery Reliability Is the Provider’s Problem

Consumer endpoints are flaky by nature — deploys, DNS hiccups, reboots. Providers must treat each delivery as an unreliable operation:

 delivery pipeline per event:
   event → durable queue → attempt POST
              ▲                  │
              └── retry on failure with backoff
                  e.g. 1m, 5m, 30m, 2h, ... up to ~24h
 
 at-least-once semantics → consumers MUST handle duplicates
 (event id + idempotent processing, same as any queue consumer)

Security: Verify Everything

 the endpoint is PUBLIC — anyone can POST fake "payment succeeded"

 signing (HMAC):
   signature = HMAC-SHA256(secret, timestamp + "." + body)
   header carries timestamp + signature
   consumer recomputes; mismatch → reject

 replay protection:
   reject timestamps older than ~5 minutes;
   attacker can't replay an old captured valid request

 secret rotation: dual-secret grace window during rotation

Skipping verification is how fraudulent “refunds” happen. This is the single most critical implementation detail of webhooks.

Consumer-Side Discipline

Rule	Why
Ack fast (2xx), process async	Slow handlers cause provider timeouts → spurious retries
Idempotent on event id	At-least-once means duplicates arrive
Handle out-of-order	Retries + parallelism scramble arrival
Return 4xx for permanent rejection	Tells provider to stop retrying

The ack-fast rule surprises people: enqueue internally and return immediately; heavy work happens after the HTTP response.

Polling Fallback

Firewalls and corporate proxies sometimes block inbound. Mature platforms offer both: webhooks primary, a fetch-events API (“replay missed”) as fallback and reconciliation tool.

Interview Framing

“Notify merchants when trips complete” invites webhook design. Scored shape: registration flow, signed delivery with HMAC + timestamp, durable retry-with-backoff queue, at-least-once consequences for consumers, ack-fast guidance. Candidates who forget signature verification get the security follow-up they’ve earned.