Designing a send queue that does not send twice
A send queue has one job that is hard: deliver exactly once, from a system that will fail halfway. The two mechanisms that get you there are an idempotency key enforced in the database and a state machine that only ever moves forward.
The states
Keep them few and keep them ordered. Every additional state is another transition somebody has to reason about at 3am.
queued -> sending -> sent -> delivered -> read
\
-> failedThe rule that matters more than the diagram: never move backwards. A late-arriving sent callback must not overwrite delivered. If a retry loop selects rows in sending and a status update can put a delivered message back there, the loop will re-send messages the customer already has - and nothing in the database will look wrong afterwards.
Idempotency, properly
The caller supplies a key; you enforce it with a unique index. Not a check-then-insert, which two concurrent requests both pass.
CREATE UNIQUE INDEX messages_idem
ON messages (tenant_id, idempotency_key)
WHERE idempotency_key IS NOT NULL;On conflict, return the message that already exists rather than an error. A retrying client wants to know the state of its send, and handing it a 409 makes the sensible client retry again.
Which failures may be retried
| Failure | Retry? | Why |
|---|---|---|
| Network timeout | Yes, with backoff | It may have worked; check state first |
| 429 or provider rate limit | Yes, after the stated delay | Temporary by definition |
| 5xx from the provider | Yes, with backoff and a cap | Their problem, probably passing |
| Invalid recipient | No | It will fail identically forever |
| Window closed | No | Only a new inbound message changes it |
| Template rejected | No | Needs a human to fix the template |
| Token invalid | No | Needs a reconnection, not a retry |
A timeout is the dangerous one: you do not know whether it arrived. Retrying blind duplicates; not retrying loses. This is exactly what the idempotency key is for - retry with the same key, and let the store decide whether it is a new send.
Pacing, and why it belongs in the queue
Sending as fast as the API allows is both a ban signal and a self-inflicted rate limit. Pace per sending identity rather than globally: two numbers should not share one budget, or one busy campaign starves the other's replies.
Give replies to live conversations a separate, faster lane. A customer waiting for an answer should never be behind four thousand campaign messages, and the simplest way to guarantee that is to not put them in the same queue.
What to store per message
- The idempotency key, and the provider's own id once you have it.
- Attempt count and the time of the last attempt.
- The last failure, verbatim, including the provider's code and subcode. Paraphrasing an error destroys the only thing that identifies it.
- Which identity it went out from - the number or Page - because that is what a reply comes back to.
- Timestamps for every state change, not just the current state.
The failure that survives all of this
A queue that stops draining, silently. Nothing errors because nothing runs. Alert on the age of the oldest queued message rather than on error rate: it is the one signal that fires for a stuck worker, a lost lock, a crashed consumer and a full disk, all of which look identical from the outside.
Questions people actually ask
How should I retry a failed message?
Only for temporary failures, with exponential backoff and jitter, and with a cap. Permanent failures - invalid recipient, closed window, rejected template - must not be retried at all, because the second attempt fails identically.
How do I stop duplicate messages?
An idempotency key supplied by the caller and enforced by a unique index, plus a state machine that never moves a message backwards. Either alone will let duplicates through.
Njiwa
One inbox for WhatsApp, Facebook and Instagram, and an API that sends over either WhatsApp rail. 7 days free, no card.