Every job runs, and its effect lands once.
Background work sounds simple until the job that ran twice and charged the customer twice, or the one that vanished when a worker crashed mid-run. True exactly-once delivery is a myth in a distributed system, so we build the thing that actually works: at-least-once delivery with idempotent handlers, so a job can be retried safely and its effect still happens exactly once.
Drive it: enqueue a charge, then hit Duplicate it, the same job delivered twice, and watch the second one get deduped instead of charging again. The flaky job retries on its own.
duplicate the charge: same idempotency key, so it's acked and skipped, never charged twice
A job that's safe to run twice.
What you just operated is this. The first gate is the whole trick: the idempotency check turns at-least-once delivery, the only kind you can actually guarantee, into an exactly-once effect. Past it, a worker leases the job, does the work, and either finishes or retries into a dead-letter queue.
Before any work, the job's key is checked. Seen before, it's acked and skipped, so a retry does nothing twice.
A free worker leases the job with a visibility timeout, so a crashed worker's job returns to the queue, not the void.
The handler does the one real effect and records the key atomically, so the two can't drift apart.
A failure retries on exponential backoff, so a flaky dependency recovers instead of losing the job.
After the last attempt, the job parks with its payload and error, for a fix and a replay.
Exactly-once is a handler, not a promise.
No queue can truly deliver a message exactly once, network and crashes make sure of that. So the honest design doesn't pretend to. It delivers at-least-once and makes the handler idempotent, so a retry is a no-op. The check and the work commit together, so a crash between them can't leave the effect done but unrecorded.
1// worker.ts — at-least-once delivery, exactly-once effect2export async function handle(job) {3 if (await store.seen(job.idempotencyKey)) // ran before?4 return ack(job); // yes: no-op, ack the retry56 await withRetry({ backoff: "exponential" }, async () => {7 await doWork(job.payload); // the real, single effect8 await store.markSeen(job.idempotencyKey); // atomic with the work9 });1011 return ack(job);12}The only guarantee a queue can actually make. A job is retried until a worker acks it, so it's never lost.
The part you write: a re-run detects it already happened and does nothing. That's what the key is for.
Delivery you can trust plus a handler you can retry gives the thing everyone actually wanted: the effect happens once.
The properties that make it dependable.
A queue is easy to add and easy to trust too much. These are the guarantees that make background work something you can actually build a business process on.
At-least-once delivery plus idempotent handlers, so a retried job never doubles a charge, an email, or a write.
A transient failure backs off and tries again, so a flaky dependency heals itself instead of dropping work.
Jobs for the same entity run in sequence, without serializing the whole queue and killing throughput.
A worker leases a job with a visibility timeout, so if it dies mid-run the job returns to the queue, not the void.
A job that exhausts its retries parks with its payload and error, so you fix the cause and replay, not reconstruct it.
Per-queue worker counts and rate limits, so a burst of jobs can't overwhelm the database or a downstream API.
Anyone selling you true exactly-once delivery is selling you something that doesn't exist.
Networks drop acks and workers crash at the worst moment, so a message will sometimes arrive twice, no matter what the queue promises. Pretending otherwise is how you end up double-charging a customer. The honest, robust answer is the opposite: assume every job can be delivered more than once, and make the handler not care. Deliver at-least-once, dedupe on a key, commit the work and the record together. We build it that way because it's the version that survives contact with production.
Background work that lands once, even when it's delivered twice.
At-least-once delivery with idempotent handlers, retries with backoff, per-key ordering, and a dead-letter queue, so a job is safe to run twice and its effect happens once.