Design Webhook Delivery
You're asked to design a webhook delivery platform like those used by Stripe, GitHub, or Shopify. Product events become HTTPS requests to customer endpoints. Those endpoints run code you cannot see and fail in ways you cannot control, so the platform must deliver reliably without letting one bad endpoint delay everyone else. Let's treat this small HTTP feature as a reliability problem.
Clarify the requirements
The platform emits events such as payment.succeeded and order.shipped, and customers register HTTPS endpoints for the event types they want. Clarify the contract before designing the system:
- Delivery guarantee? At-least-once. Customers must expect duplicates, which shapes the payload design (event IDs for dedup).
- Ordering? Offer best-effort ordering at most. An endpoint that times out makes global ordering impractical, so set that expectation early.
- Latency? Target seconds rather than milliseconds. Because delivery is asynchronous, you can use the full jobs toolkit.
- What does the customer see? Delivery logs, attempt history, manual redelivery, and endpoint health. The dashboard is part of the product.
- Security? Customers need to verify that requests came from the platform, and a malicious endpoint URL must not expose internal services.
The platform must not lose an event after acknowledging it internally. One slow endpoint must not delay another customer's deliveries, and an endpoint can remain unavailable for a week without threatening the platform.
Back-of-envelope numbers
Assume 50k active endpoints and 200M events/day, averaging two endpoint subscriptions per event.
- Deliveries: average, so size the dispatch path for a 10× peak near
- Attempt logs: at full retention
- Retries: if 5% of endpoints are failing at any moment, retry traffic adds a large and bursty multiplier on top of all of the above
Call out retry amplification when you discuss capacity. If a large customer's endpoint goes down, load increases because every failed delivery creates scheduled work for the future. Plan capacity for those retries as well as first attempts.
High-level architecture
- Product services. Emit events as part of their own transactions.
- Outbox. Commits the event row with the business change, so an acknowledged event cannot be lost before delivery starts.
- Fan-out. Matches each event against subscriptions (customer, endpoint, event-type filters) and creates one delivery job per matching endpoint.
- Per-endpoint queues. Isolate failures by serializing deliveries, or allowing only narrow concurrency, for each endpoint. Other endpoints do not share its backlog.
- Delivery workers. Pull jobs, sign the payload, POST with a timeout, classify the response, and schedule retries with backoff.
- Attempt log. Stores the timestamp, status code, and latency for each attempt. It supports the customer dashboard, replay, and endpoint health scoring.
A delivery fits the job model from async jobs and workers: it has durable state, an attempt counter, a retry budget, and a terminal status. Use those familiar terms to explain the design clearly.
Deep dive 1: isolating endpoints
The main failure mode is a large customer's endpoint returning 429s or hanging while every other customer's deliveries wait behind it. A single global delivery queue allows this to happen.
Use one queue and concurrency cap per endpoint. This applies per-tenant fairness, with each customer endpoint treated as a tenant. Set a small concurrency cap, often 1 to preserve rough ordering, and give each endpoint a token-bucket rate limit that honors Retry-After. Add a circuit breaker that trips after N consecutive failures. Once tripped, the breaker stops workers from repeatedly calling a dead endpoint. Deliveries remain in that endpoint's queue while periodic probes check for recovery.
Deep dive 2: retries, the DLQ, and replay
Classify responses before retrying them. Timeouts, 429s, and 5xx responses are retryable. A 400 means the endpoint rejected the payload, while a 410 means it wants delivery to stop; retrying either wastes capacity and clutters the customer's logs. Use exponential backoff with jitter over a long window, such as 8 attempts across 24 hours, because endpoint outages may last minutes or hours.
After the retry budget is spent, move the delivery to a dead-letter state. Show it in the dashboard, retain it for a window such as 30 days, and allow replay. A customer may replay one event or request everything since an outage began. The second case requires a rate-limited bulk backfill so the recovered endpoint is not overwhelmed. Both paths depend on stored payloads and the attempt log. Every payload carries a stable event_id, and customers make their handlers idempotent so duplicate deliveries are safe. This applies delivery semantics and idempotency to the customer's side of the connection.
Deep dive 3: security at the edge
Defend both sides of the connection. For outbound authenticity, sign the body and a timestamp with an HMAC using a per-endpoint secret. Send the signature in a header that the customer verifies. The timestamp prevents replay of captured requests. During secret rotation, use a dual-signing window so customers can change keys without losing deliveries.
For inbound danger, treat the endpoint URL as attacker-controlled input. A malicious customer could register http://169.254.169.254/, which points to cloud metadata, or use an internal hostname. A delivery worker that sends requests there creates an SSRF vulnerability. Resolve the destination and reject private IP ranges at request time, not only at registration, because DNS can change. Route traffic through a dedicated NAT with no internal routes, and do not follow redirects blindly. Include SSRF when you discuss endpoint validation.
Deep dive 4: what "ordering" honestly means
Strict delivery order is difficult to guarantee at scale. A retrying event falls behind newer ones unless the system fully serializes the endpoint's queue. Full serialization makes one slow event block everything behind it.
Use per-endpoint FIFO with concurrency 1 for rough ordering on the happy path. When an event moves to its retry schedule, release the queue so newer events can continue. Include the creation timestamp and a per-resource sequence number in each payload so the consumer can reorder events or discard stale updates. Explain that best-effort ordering avoids coupling every event's latency to the slowest retry.
Leveling signals
Related lessons
The mirror image of this problem: there you protect other people's servers from your traffic, here you protect your platform from other people's failures.
Fire recurring jobs on time across a fleet, with the same lease, retry, and dead-letter machinery.