Part 3 · System design · Design problems
Notification and webhook delivery
When something happens, such as a job being assigned or a payment settling, notify customers by webhook, email and push. Customer endpoints are slow and sometimes down.
Ask first
- Which channels? Does order matter per customer?
- Is at least once fine? (Exactly once over HTTP is not possible.)
- How long do we keep retrying?
- Do customers need to see and replay failed deliveries?
Numbers
| events | say 5M a day, about 60 per second, peaks 10x |
| slow endpoints | a 10 second timeout at 600 per second means 6,000 requests in flight, so use async workers, not request threads |
| retention | keep 30 days of attempts for the replay screen |
API
POST /webhook-endpoints { url, events[] } { id, secret }
Delivery: POST customer_url
Webhook-Id: evt_123
Webhook-Signature: t=1700000000,v1=<hmac>
{ id, type, created, data }Data
outbox(event_id, type, payload, created_at) deliveries(event_id, endpoint_id, attempt, status, next_attempt_at) endpoints(id, url, secret, disabled_at)
Deep dive: Where do events come from?
A transactional outbox, written in the same transaction as the business change and published by a relay. No lost events.
Retry schedule?
Exponential backoff with jitter spread over about a day, then a dead letter queue. Retry timeouts, 408, 429 and 5xx. Other 4xx means the request itself is wrong: stop.
Can one slow customer slow everyone?
Not if each endpoint has its own queue or concurrency limit. Endpoints that fail for days get disabled and the owner gets an email.
How do customers trust it?
An HMAC signature over timestamp and body, a constant time compare, and old timestamps rejected so captured requests can't be replayed. They dedupe on the event id, because retries mean duplicates.
See the core idea run
Core logic (JavaScript, tested)
// 2xx = done. 408, 429, 5xx and timeouts (status 0) may pass, so retry. Other 4xx is their bug: stop.
const retryable = status => status === 0 || status === 408 || status === 429 || status >= 500;
function deliver(event, send, { maxAttempts = 5, delay = a => backoff(a) } = {}) {
const log = [];
for (let attempt = 0; attempt < maxAttempts; attempt++) {
// ponytail: real workers re-enqueue with a delay instead of waiting in a loop
const waitBefore = attempt ? delay(attempt - 1) : 0;
const status = send(event, attempt);
log.push({ attempt, status, waitBefore });
if (status >= 200 && status < 300) return { result: "delivered", log };
if (!retryable(status)) return { result: "dead-letter", reason: `permanent ${status}`, log };
}
return { result: "dead-letter", reason: "out of attempts", log };
}
// Sign timestamp + body, so receivers can check it came from us and isn't an old replay.
const sign = (secret, ts, body) => createHmac("sha256", secret).update(`${ts}.${body}`).digest("hex");
function verify(secret, header, body, nowSec, toleranceSec = 300) {
const { t, v1 } = Object.fromEntries(header.split(",").map(p => p.split("=")));
if (!t || Math.abs(nowSec - Number(t)) > toleranceSec) return false; // too old: maybe a replay
const want = Buffer.from(sign(secret, t, body)), got = Buffer.from(v1 ?? "");
return got.length === want.length && timingSafeEqual(got, want); // constant time compare
}Failures
| Customer endpoint down | Backoff, then the dead letter queue, then a replay button. |
| Our worker dies mid send | The message reappears after its visibility timeout. At least once; the customer dedupes. |
| Email provider outage | The breaker opens, the queue grows, and it drains when the provider recovers. |
Trade-offs
- At least once plus dedupe beats chasing exactly once.
- Strict order per customer costs throughput; most receivers only need ids and timestamps.
Say out loud: Outbox into a queue. Workers send signed requests with exponential backoff and jitter, and give up into a dead letter queue with replay. Each endpoint is isolated, so one dead customer can't starve the rest. Delivery is at least once, so every event has an id to dedupe on.