Idempotency in Distributed Systems

Distributed systems are unreliable. Requests, responses, and messages can be lost, delayed, or interrupted. As a result, a caller cannot always determine whether an operation completed successfully.

Imagine a client sends a request to create a payment. The server successfully transfers the money and commits all changes to the database. However, the network connection is lost before the response reaches the client.

From the client’s perspective, there are several possibilities:

• the request never reached the server;
• the server started processing but failed;
• the operation completed successfully, but the response was lost.

The client has no way to know which one actually happened. The safest option is to retry the request.

The problem is that retrying the request may execute the same logical operation twice. For a payment, that could mean transferring money twice. For an order, it could create two identical orders. For a message consumer, it could process the same message multiple times.

This is exactly the problem that idempotency solves.

What is idempotency?

Idempotency makes retries safe.

When a client isn’t sure whether the server successfully processed a request, it can retry the same request using the same idempotency key.

The server recognizes that both requests represent the same logical operation, executes it only once, and returns the same response that it returned for the original request.

From the client’s point of view, retrying the request produces exactly the same result as if the first response had been delivered successfully.

How does it work?

The client generates a unique idempotency key and sends it together with the request.

POST /payments
Idempotency-Key: 9d88c40b-66d4-4c0d-a379-1d8b4ef2d6a2

When the server receives the request, it starts a database transaction and:

1. Executes the business operation.
2. Stores the idempotency record together with the response.
3. Commits the transaction.

The business data and the idempotency record must be committed atomically. Otherwise, the server could successfully execute the operation but fail before storing the idempotency record, allowing a retry to execute the same operation again.

When another request arrives with the same idempotency key, the server does not execute the business logic again. Instead, it returns the response that was stored during the original execution.

Why can’t the server simply detect duplicate requests?

A common question is why the server cannot simply compare request payloads.

Imagine a user transfers $100 to the same account today and then sends another transfer with exactly the same information tomorrow.

Although both requests have identical payloads, they represent two different business operations.

Only the client knows whether a request is a retry or a completely new operation. That is why the client, not the server, generates the idempotency key.

The key identifies a single logical operation, not a particular request.

Where is idempotency used?

Idempotency is useful whenever an operation may be retried.

Common examples include:

• payment processing;
• order creation;
• invoice generation;
• webhook handling;
• REST APIs;
• Kafka consumers;
• RabbitMQ consumers;
• background jobs;
• Saga steps.

The caller may be an HTTP client, a background worker, or a message consumer. In every case, if it cannot determine whether the previous attempt succeeded, it may retry the operation. Idempotency guarantees that retrying the same logical operation does not produce additional side effects.

Common implementation approaches

There are several ways to implement idempotency.

Separate idempotency table

The most common approach is storing processed operations in a dedicated table.

The table typically contains:

• idempotency key;
• operation status;
• stored response;
• creation time;
• expiration time.

When a retry arrives, the server finds the existing record and returns the stored response.

Business table with a unique constraint

Sometimes the idempotency key is stored directly in the business table and protected by a unique constraint.

For example, a Payments table may contain an IdempotencyKey column with a unique index. Attempting to create another payment with the same key will fail, allowing the server to return the existing result instead of creating another payment.

Common mistakes

Treating idempotency as duplicate detection

Idempotency does not prevent users from intentionally performing the same action multiple times.

If a client sends two requests with different idempotency keys, they represent two different logical operations, even if the payload is identical.

Saving only the idempotency key

In many cases, storing only the key is not enough.

To return exactly the same response, the server usually stores the response together with the idempotency record.

Non-atomic persistence

The idempotency record and the business operation must be committed in the same transaction.

Otherwise, partial failures may allow the same operation to be executed multiple times.

Summary

Idempotency exists because retries are unavoidable in unreliable systems.

Instead of trying to detect duplicate requests, the client assigns a unique idempotency key to a logical operation. The server stores that key together with the result of the operation and guarantees that every retry with the same key returns the original response instead of executing the operation again.

In other words, idempotency doesn’t eliminate retries—it makes them safe.