Delivery Guarantees & Idempotent Consumers

Delivery Guarantees and Consumer Idempotency

Every messaging system eventually delivers a message twice. A consumer processes an order and crashes before acknowledging it; the broker redelivers. A producer times out waiting for confirmation and retries a send that actually succeeded. A network partition makes a consumer group rebalance mid-batch. None of these are bugs — they are the unavoidable consequence of sending messages over unreliable networks between processes that can fail at any moment.

The practical answer is not to search for a perfect "exactly-once" transport. It is at-least-once delivery combined with idempotent consumers: make processing the same message twice produce the same result as processing it once. This page explains the guarantees and the patterns that make consumers safe.

TL;DR

At-most-once: never duplicated, may be lost. At-least-once: never lost, may be duplicated. Exactly-once: the effect happens once.
End to end, exactly-once delivery is not achievable in general; exactly-once processing is achieved with at-least-once plus idempotency.
Record processed message IDs in the same transaction as the side effect, or make updates naturally idempotent.
Use versions or sequence numbers to ignore stale and out-of-order updates.
Pass idempotency keys to external APIs (payments, email) so retries do not repeat effects.
Make producers reliable with the outbox pattern, and route poison messages to dead-letter queues.

Quick Example

A consumer that debits an account exactly once per message, even when the message arrives twice:

The deduplication record and the balance change commit together. If the process crashes before the commit, neither happened and the redelivery does the work. If it crashes after the commit but before the acknowledgment, the redelivery finds the record and does nothing.

Core Concepts

The Three Guarantees

Why "Exactly-Once" Has a Boundary

Kafka's transactions give exactly-once semantics for read-process-write within Kafka: consumed offsets and produced records commit atomically. Stream processors such as Flink extend this with checkpoints and transactional sinks. But the moment a consumer charges a card, sends an email, or calls another service, that side effect is outside the transaction. Each such effect needs its own idempotency. See Kafka Exactly-Once.

Sources of Duplicates

Consumer crashes or times out after processing, before acknowledging.
Visibility timeouts (SQS) or acknowledgment deadlines (Pub/Sub) expire during slow processing.
Consumer group rebalances reassign partitions before offsets are committed.
Producers retry sends after ambiguous failures.
Operators replay messages from dead-letter queues or rewind offsets.

Idempotency Patterns

Processed-Message Table

Store (consumer, message_id) with a unique constraint in the same database transaction as the side effect, as in the example above. Keep entries at least as long as duplicates can arrive — the broker's redelivery window plus any replay window — and expire them afterwards.

Natural Idempotency

Design operations so repeating them is harmless:

Versioned Updates

When messages carry a version or source sequence number, apply them only if they are newer than what is stored. This handles duplicates and out-of-order delivery:

Idempotency Keys for External Calls

Payment providers and many APIs accept an idempotency key; repeated requests with the same key return the original result instead of repeating the effect. Derive the key from the message ID so every retry of that message uses the same key. See Idempotency.

Reliable Producers: The Outbox

Publishing a message and updating a database are two separate systems. Writing the business change and an outbox row in one transaction, then relaying outbox rows to the broker, ensures every committed change produces a message — at least once. Consumers then deduplicate by the outbox row's ID. See Outbox Pattern.

Ordering and Retries

Brokers usually preserve order only within a partition or queue; retries and parallel consumers break it. Use versioned updates where order matters.
Retry with backoff for transient failures, but cap attempts and move poison messages to a dead-letter queue so they do not block others. See Dead-Letter Queues and Replay.
When replaying from a dead-letter queue, idempotency is what makes replay safe.

Best Practices

Give Every Message a Stable ID

Assign IDs at the source — the outbox row ID, an event UUID — not at publish time, so retries of the same logical message share an ID.

Acknowledge After Commit

Acknowledge or commit offsets only after the side effect is durable. Acknowledging first trades duplicates for data loss.

Keep the Dedupe Check and the Effect Atomic

A deduplication check in Redis followed by a database write in a separate step leaves a window where a crash causes either loss or duplication. Put both in one transaction where possible.

Test Duplicates and Reordering

Write tests that deliver the same message twice and deliver messages out of order. These are the cases that production will produce.

Common Mistakes

Checking, Then Acting, Without Atomicity

Trusting "Exactly-Once" Marketing

Dedupe Records That Expire Too Soon

FAQ

What is the difference between at-least-once and exactly-once?

At-least-once guarantees a message is not lost but may be processed more than once. Exactly-once means its effect happens once. In practice, exactly-once processing is built from at-least-once delivery plus idempotent consumers or transactional pipelines.

Is exactly-once delivery possible?

Not across arbitrary systems over unreliable networks. Exactly-once processing is achievable within a transactional boundary, such as Kafka transactions or a database that records message IDs alongside effects.

What is an idempotent consumer?

A consumer whose processing of the same message more than once leaves the system in the same state as processing it once — through deduplication records, upserts, versioned updates, or idempotency keys.

How long should I keep processed message IDs?

At least as long as duplicates can arrive: broker redelivery windows, consumer lag, and any replay window from dead-letter queues or offset resets. Natural idempotency avoids the question entirely.

Do FIFO queues remove the need for idempotency?

No. FIFO queues such as SQS FIFO deduplicate within a limited window and preserve order within message groups, but consumers can still process a message and fail before deleting it. Idempotent processing remains necessary.