Webhook Delivery System
Reliably call customer endpoints: at-least-once delivery over a durable queue, backoff-and-jitter retries, a dead-letter path with replay, signed payloads for receiver idempotency, and per-endpoint circuit breakers.
~30 min · intermediate
Problem & Requirements
Notify customers of events by calling HTTP endpoints they register. When something happens — a payment succeeds, a build finishes — the system POSTs a payload to the customer's URL, and it has to get there even though that URL is outside your control and fails in every way an HTTP call can: timeouts, 500s, TLS errors, DNS failures, endpoints that are down for hours, endpoints that accept the request but process it twice. The job is to deliver reliably to infrastructure you don't own and can't fix.
That premise is what separates this from an ordinary outbound call. Because the endpoint is unreliable, delivery has to be asynchronous and durable, and the only guarantee worth offering is at-least-once: the event will arrive, possibly more than once. Transient failures get retries with backoff and jitter so a flapping endpoint isn't hammered. Permanently failing deliveries land in a dead-letter path that the customer can inspect and replay. At-least-once is only safe because the producer signs each payload with a stable event id, turning idempotency into a contract the receiver can honor. And an endpoint that's consistently broken gets a circuit breaker so the worker fleet stops wasting capacity on it and gives it room to recover. Those five principles are the prototype. (It shares the queue/retry/DLQ spine with the email-delivery prototype; the parts unique to webhooks are the signing contract and the breaker.)
Functional
- Register an endpoint with a secret; on an event, deliver the signed payload to that endpoint.
- Guarantee at-least-once delivery, with retries on transient failure and a dead-letter path on exhaustion.
- Let a consistently failing endpoint stop receiving attempts temporarily, and recover automatically.
Non-functional (back-of-envelope)
| Quantity | Target | What it forces |
|---|---|---|
| Event volume | 10⁶+ /day across many endpoints | Async delivery on a worker fleet; never POST inline with the triggering action |
| Attempt timeout | ~10–30 s | A slow endpoint can't block a worker indefinitely; bounded per-attempt timeout |
| Delivery semantics | at-least-once | Receiver must dedupe; producer must send a stable event id every retry |
| Retry window | up to ~3 days | Long backoff curve, bounded by attempts and total age (Stripe-style) |
| Endpoint reliability | highly variable | One dead endpoint must not starve healthy ones → circuit breaker + isolation |
| Replay | on demand | Dead-lettered or past events re-deliverable from a dashboard |
The reliability row is the one that shapes the back half. A single customer endpoint that hangs for thirty seconds on every request will, without protection, tie up workers retrying it and slow delivery to everyone else. The circuit breaker plus per-endpoint isolation is what contains that blast radius, and it's the part with no analog in the email prototype.
Design
Six components, each tied to its principle:
- Dispatcher — on an event, builds a delivery job (stable event id, endpoint id, payload) and enqueues it. The event id is assigned once and reused across every retry, which is what makes receiver dedup possible.
- Durable queue — holds jobs and supports delayed re-enqueue for backoff. At-least-once by design: a worker that dies after sending but before acknowledging causes a redelivery, which the receiver must tolerate. See at-least-once delivery.
- Delivery workers — sign the payload, POST it with a bounded timeout, and treat a 2xx as success and anything else (non-2xx, timeout, connection error) as a failure to retry.
- Retry scheduler — re-enqueues a failed job with full-jitter exponential backoff, bounded by attempts and age. See retries with backoff.
- Dead-letter queue — receives jobs that exhaust their budget, where they're stored for inspection and replay rather than dropped. See dead-letter queues.
- Signer + per-endpoint circuit breaker — the signer HMACs each payload with the endpoint's secret over a timestamp and the body, and includes the event id, so the receiver can verify authenticity and dedupe — the idempotency contract. The breaker tracks failures per endpoint and, past a threshold, opens to fail-fast and park further attempts until a cooldown lets it probe recovery. See circuit breakers.
Two real systems make this concrete and agree on the shape. Stripe signs webhooks with an HMAC in Stripe-Signature over a timestamp and the raw body, gives each event a stable id for idempotent processing, retries with exponential backoff for up to roughly three days, and lets you replay events from the dashboard. GitHub signs with X-Hub-Signature-256, identifies each delivery with X-GitHub-Delivery for dedup, and supports manual redelivery. Both push the same contract onto the receiver — verify the signature, dedupe on the id — which is the only way at-least-once delivery is safe to offer.
Build it
The starting point calls the customer endpoint inline, inside whatever action triggered the event. It works when the endpoint is healthy and fast, and it fails the moment it isn't: a slow endpoint blocks the triggering action, a timeout or 500 loses the notification, and the customer's downtime becomes your downtime. The unreliability of the endpoint is exactly what this couples to, and every step removes one consequence of that coupling.
Decouple the event from its delivery. The dispatcher enqueues a durable job and returns; a separate worker pulls jobs and POSTs them, treating a 2xx as the only success. The queue is at-least-once: if a worker crashes after the POST but before acknowledging, the job is redelivered and the endpoint is called again. That guarantee is deliberate — building exactly-once across a network you don't control is impractical, so the design promises at-least-once and makes duplicates safe later, in the idempotency step.
Any non-2xx, timeout, or connection error is a failure to retry, and retrying immediately just batters an endpoint that's already struggling. Re-enqueue with a delay that grows exponentially per attempt and apply full jitter — a uniform random pick in [0, window] — so a backlog of jobs for the same endpoint doesn't retry in a synchronized wave. The curve is capped. This is retries with backoff and jitter; full jitter is the variant that spreads retries most evenly across the fleet.
Bound it and keep it: the dead-letter queue
Retries can't run forever — a permanently broken endpoint would loop until the heat death of the worker pool. Cap on attempts and total age, and route an exhausted job to a dead-letter queue instead of dropping it. Unlike a lost packet, a dead-lettered webhook is a customer-visible artifact: it's stored with its failure reason so the customer can see what didn't deliver and trigger a replay once they've fixed their endpoint. The DLQ is what makes "we tried and gave up" a recoverable state.
At-least-once means the receiver will sometimes get the same event twice, so the producer's job is to make that harmless. Send the same event id on every attempt and sign the payload with the endpoint's secret over a timestamp and the raw body, so the receiver can verify the request is genuinely yours and skip an id it has already processed. This is the idempotency contract: the producer guarantees a stable id and at-least-once delivery; the receiver dedupes on the id to get effectively-once processing. The timestamp in the signature also lets the receiver reject stale replays.
Retries and isolation still aren't enough on their own — an endpoint that's been down for an hour will, for every event, burn a worker on a full timeout before failing. Track outcomes per endpoint and, once consecutive failures cross a threshold, open the breaker: stop attempting and park the endpoint's jobs for a cooldown, failing fast instead of timing out. After the cooldown, let a single probe through (half-open); success closes the breaker, failure re-opens it. This is a circuit breaker per endpoint — it protects the worker fleet from wasting capacity and gives the failing endpoint room to recover before the next wave.
Tradeoffs
| Decision | What it buys | What it costs |
|---|---|---|
| Async over a durable queue | Triggering action never blocks on customer uptime; delivery scales out | Delivery is eventual; a queue to persist and operate |
| At-least-once semantics | Crash-safe; simple to reason about | Duplicates, so the receiver must dedupe — pushed onto the customer |
| Backoff + full jitter | Spares struggling endpoints; de-synchronizes retry waves | Late delivery; ordering across retries is lost |
| Dead-letter queue + replay | Failures are visible and recoverable, never silently lost | Storage and a replay surface; dead-lettered events need customer action |
| Signed payload + stable id (idempotency) | Receiver can authenticate and dedupe → effectively-once processing | Secret management and rotation; the contract only helps receivers who honor it |
| Per-endpoint circuit breaker | Stops wasting workers on dead endpoints; gives them recovery room | Tuning threshold/cooldown; an open breaker delays delivery to a recovering endpoint |
Scaling it up
The toy leaves out the parts that matter most under real traffic. The notable gaps:
Per-endpoint isolation, not just a breaker. A shared worker pool still lets one slow endpoint occupy workers up to its timeout. Production systems isolate per endpoint or per customer — separate queues or partitions, per-endpoint concurrency caps, sometimes a bulkhead so a noisy customer can consume only its share — so a backlog for one destination can't starve the rest. The breaker reduces the damage; isolation bounds it.
Ordering, which at-least-once quietly breaks. Retries and parallel workers deliver events out of order, and some consumers care (a created then updated for the same resource). Answers include per-resource partitioning so one resource's events deliver in sequence, sending sequence numbers the receiver can reorder on, or simply documenting that order isn't guaranteed — which is what most providers do, including Stripe.
Signing as a real protocol. A single HMAC secret per endpoint is the floor. Real systems support secret rotation (accept two valid secrets during a window), versioned signature schemes, and a replay-protection window on the timestamp. The emerging Standard Webhooks spec codifies exactly the Webhook-Id/Webhook-Timestamp/Webhook-Signature shape used above, which is worth adopting rather than inventing.
The dead-letter path is a product surface. "Stored and replayable" implies a dashboard showing recent deliveries, response codes and bodies, and one-click replay — what Stripe and GitHub both ship. That turns the DLQ from an ops artifact into a customer self-service tool, which is most of its value.
Backoff and breakers informed by signal, not just counts. A smarter retry uses the response — honoring Retry-After, backing off harder on 429s, treating a 410 Gone as permanent and disabling the endpoint outright. The breaker likewise can key on error rate over a window rather than a raw consecutive-failure count, which behaves better under intermittent failures.
This is the second messaging-and-reliability prototype, and it shares its spine with the email-delivery one: both are accept-queue-retry-DLQ pipelines, differing in transport (HTTP vs SMTP) and in the webhook-specific signing contract and circuit breaker. The natural next steps are a dedicated durable queue prototype — the partitioned, persistent, at-least-once log both of these assume — and an idempotency / dedup store prototype that builds the receiver-side and producer-side dedup properly. A rate limiter prototype also plugs in directly here, to shape per-endpoint delivery concurrency. Each reuses this foundation set without re-covering the delivery semantics settled here.
References
- Stripe webhooks — signing, event ids for idempotent handling, retry schedule, and dashboard replay: https://docs.stripe.com/webhooks
- GitHub webhooks —
X-Hub-Signature-256signing,X-GitHub-Deliveryids, and redelivery: https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries - Standard Webhooks — an open spec for the signing and id headers used in step 5: https://www.standardwebhooks.com/
- Nygard, Release It! — the circuit breaker pattern and stability patterns it belongs to.
- AWS, Exponential Backoff and Jitter — why full jitter spreads retries best: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
- Foundations referenced inline: at-least-once delivery · retries with backoff · dead-letter queues · idempotency · circuit breakers