A useful read should leave you with something you can try, not just something you can repeat.
01 / Assume repetition
Assume every event can arrive twice.
Webhook providers retry. Networks time out. A user clicks again. Your system should recognize the same job instead of charging twice or publishing twice.
Give every provider event a stable event ID and store it with a unique constraint, for example a primary key on (provider, event_id). Verify the provider signature before accepting the event. In one database transaction, record the event and create the job; if the key already exists, return the prior result. This is idempotency: repeating the request has one durable effect.
Do not ask a model to decide whether an event is a duplicate. That is deterministic infrastructure work. Keep a job row with explicit status, attempt count, next retry time, and a concise error code so the app can recover and an operator can understand what happened.
- Verify signature → persist event ID → enqueue once.
- Unique constraint is the race-safe duplicate guard.
- Keep event receipt and job state; avoid relying on in-memory flags.
02 / Recover deliberately
Retry the failure, not the side effect.
A timeout does not tell you whether the other service completed the work. Retrying blindly can duplicate an irreversible action.
Use a persistent queue or job table, bounded exponential backoff with jitter, and a maximum attempt count. Retry transient rate limits and server errors according to provider guidance; do not retry invalid input or permission failures as if waiting would fix them. If the model or external API supports an idempotency key, send the same durable job key on every attempt.
When the retry budget is exhausted, move the job to a dead-letter or needs-attention state with a safe operator retry path. Exactly-once execution across independent systems is not a promise you can get from a single webhook handler; build for at-least-once delivery plus idempotent effects and reconcile uncertain outcomes.
- Use bounded backoff + jitter; honor Retry-After when available.
- Classify permanent versus transient errors.
- Persist attempts and give exhausted jobs a visible recovery path.
03 / Choose the approval boundary
Automate drafts. Gate consequential actions.
The more public, costly, or hard-to-reverse an action is, the more valuable a human confirmation becomes.
Let the system collect facts, draft copy, label records, and prepare a proposed action. Before it sends money, changes access, deletes data, or publishes a public claim, show the human the exact action and the evidence behind it. Record who approved it, what version they saw, and when the action occurred.
Give an operator a pause switch, an audit trail, and a safe way to resume after a provider outage. Monitor queue age, repeated failures, cost per completed job, and actions waiting for review. A trustworthy automation is not one that never fails; it is one whose failure is bounded, visible, and recoverable.
- Draft → inspect → approve → execute → record.
- Pause on uncertainty or high-impact actions.
- Alert on stuck work; make replay safe and auditable.
The happy path is only half the system.
Use a unique (provider, event_id) key. Return the already-recorded job instead of creating a second side effect.
PRIMARY KEY (provider, event_id)FINAL TEST / +25 XP
A provider call timed out after a publish request. What should the system do first?
KEEP EXPLORING / SOURCE NOTES
An original reliability guide. Provider retry semantics vary; consult the official API contract and test failure paths before automating consequential actions.
Good work should
go somewhere.
Keep the field guide close. If you want to turn a clear idea into a digital experience, take a look at what Moticulation makes.