Webhooks for messaging, and the five failures everyone hits

A messaging webhook is an HTTP POST you receive when something happens - a message arrives, a status changes. The five things that break in production are always the same: unverified signatures, replays, out of order arrival, slow handlers, and failures nobody sees. All five are cheap to fix on day one and expensive to fix after.

1. Verify the signature, on the raw body

Your webhook URL is a public endpoint that accepts data and acts on it. Without a signature check, anybody who learns the URL can tell your system a payment succeeded.

The mistake that survives code review: verifying against the parsed and re-serialised body. Key order and whitespace change, the HMAC no longer matches, and the fix people reach for is to stop verifying. Capture the raw bytes before any JSON parser touches them.

# FastAPI: the raw body, then parse - not the other way round.
@app.post("/hooks/njiwa")
async def hook(request: Request):
    raw = await request.body()
    signature = request.headers.get("X-Njiwa-Signature", "")
    expected = hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(signature, expected):
        raise HTTPException(401)
    event = json.loads(raw)
    ...

Use a constant time comparison. == on a signature is a timing oracle, and it costs nothing to avoid.

2. Assume every event arrives twice

Retries exist because networks fail. That means at-least-once delivery, which means duplicates. If your handler creates a record, sends an email, or charges anything, it will do it twice the first time your server returns a 500 after doing the work.

Every event carries an id. Store it, and make the store the guard:

seen = await db.execute(
    insert(processed_events)
    .values(event_id=event["id"])
    .on_conflict_do_nothing()
)
if seen.rowcount == 0:
    return {"ok": True}   # already handled, and that is fine

3. Do not trust arrival order

A delivered can arrive before the sent it follows, because one of them was retried. Software that assigns status on arrival will drag a message backwards through its own lifecycle.

Two defences, and use both: compare the payload timestamp against what you have stored, and only ever move a message forward through the state sequence. This is not theoretical - a delivered message dragged back to sending is exactly how a retry loop re-sends messages that already arrived, and the database looks completely normal afterwards.

4. Acknowledge first, work second

The handler's only job is to persist the event and return 200. Everything else - the database writes, the email, the third party call - happens after, on a queue you control.

Handlers that do the work inline are fine until the day a dependency is slow. Then they time out, the provider retries, the retry is also slow, and you have converted a slow database into a self inflicted denial of service.

5. Make failures visible

The worst webhook bug is the silent one: your endpoint returns 500, the provider retries for a while and gives up, and nobody notices for a week. There is no error in your logs because the requests stopped arriving.

• Log every delivery, including the ones you reject and why.
• Alert on absence - no events for an hour on a system that normally sees dozens is the signal that matters, and it is the one nothing produces on its own.
• Keep a replay tool. When you do miss a day, you want to reprocess rather than reconcile by hand.

A checklist

1. Signature verified against raw bytes, constant time.
2. Event id stored with a unique index; duplicates return 200.
3. Status moves forward only; timestamps beat arrival order.
4. 2xx returned before any slow work.
5. Endpoint responds to unknown event types without failing.
6. Alerting on silence, not just on errors.

Njiwa's webhook payloads, signature header and retry schedule are in the docs.