Idempotency Keys Are a Contract, Not a Header: Designing Retry-Safe Operations

A mobile client sends POST /orders. The server validates the request, writes the order, charges the card and starts writing the response. Somewhere between the server and the phone, a load balancer times out or the train enters a tunnel. The client sees an error and does what every resilient client is told to do: it retries. Whether the customer now has one order or two depends entirely on a decision someone made, or failed to make, when the endpoint was designed. Idempotency keys are the usual answer, but sending an Idempotency-Key header achieves nothing by itself. The header is the visible end of a contract, and the contract lives in your storage, your transaction boundaries and your downstream calls.

A timeout is not a failure, it is an unknown

The reason this matters is that a client cannot distinguish "the request never arrived" from "the request succeeded and the response was lost". Both look like a timeout or a reset connection. The only safe response to an unknown outcome is to repeat the request, and repeating is only safe if the server treats the repeat as the same request.

HTTP already has vocabulary for this. RFC 9110 defines a method as idempotent when the intended effect of several identical requests is the same as the effect of one; PUT, DELETE and the safe methods qualify, POST and PATCH do not. Two points are worth holding on to. First, this is a statement about intended semantics, not a guarantee your implementation enforces. A DELETE handler that writes an audit row and sends an email every time it is called is idempotent in the specification and not in reality. Second, most business operations that create something, such as an order, a payment or a booking, are naturally POST, which is exactly where retries hurt.

The same problem appears away from HTTP. Message brokers almost always offer at-least-once delivery, so a consumer will eventually see the same message twice, after a rebalance, a crash before acknowledgement or a visibility timeout that expired during a slow job. If you rely on queues, the design described in queue-based architecture for reliable processing only stays correct when every consumer is idempotent.

What idempotency keys actually promise

An idempotency key is a client-generated identifier for one intended operation. The contract a server makes when it accepts one has four clauses, and an implementation that skips any of them is not finished.

The first clause: the same key with the same request produces the effect once, and every repeat receives a response equivalent to the original. "Equivalent" matters. Returning 409 Already exists to a retry is a common mistake, because the client cannot tell whether its own request created the resource or someone else's did. AWS describes this in its Builders' Library article on idempotent APIs: the retry should receive a response with the same meaning as the first.

The second clause: the same key with a different request is an error, not a silent replay. If a client reuses a key for a different amount, replaying the old response hides a bug. The IETF HTTPAPI working group's draft for the Idempotency-Key header recommends 422 Unprocessable Content for this case, 409 Conflict when a request with the same key is still being processed, and 400 Bad Request when a required key is missing. As of late 2026 the document is still an Internet-Draft (revision 08), not an RFC, so treat it as a convention rather than a standard; the status codes are a sensible one to follow.

The third clause: the key is scoped. A key is unique per account, or per API credential, not globally. Otherwise two customers whose clients happen to generate the same string would collide, and worse, one could receive the other's stored response.

The fourth clause: the key has a lifetime, and the client knows it. Stripe, for example, documents that on its v1 API keys can be pruned once they are at least 24 hours old, after which a reused key is treated as a new request. Your retention must be longer than the longest retry horizon of any client, including mobile apps that queue work offline for days.

One design choice follows from the first clause: the key must come from the client, generated once per user intent and reused for every attempt of that intent. Deriving the key on the server from the request body looks tempting and is wrong, because two identical requests can be legitimately different intents, such as a customer who really does want two identical items in two separate orders.

The storage that makes the contract true

The simplest correct implementation keeps the key, a fingerprint of the request and the stored response in the same database as the business data, so that recording the key and performing the work commit or roll back together.

CREATE TABLE idempotency_keys (
  account_id     bigint      NOT NULL,
  idem_key       text        NOT NULL,
  request_hash   bytea       NOT NULL,
  state          text        NOT NULL CHECK (state IN ('started', 'completed')),
  response_code  int,
  response_body  jsonb,
  locked_at      timestamptz NOT NULL DEFAULT now(),
  created_at     timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, idem_key)
);

The handler claims the key with a write that cannot succeed twice, then either does the work or replays what is stored.

async function createOrderHandler(req: ApiRequest, db: Db): Promise<ApiResponse> {
  const key = req.header('Idempotency-Key');
  if (!key) return json(400, { error: 'Idempotency-Key header is required' });
  const hash = sha256(`${req.method} ${req.route}\n${canonicalJson(req.body)}`);

  return db.transaction(async (tx) => {
    const claimed = await tx.query(
      `INSERT INTO idempotency_keys (account_id, idem_key, request_hash, state)
       VALUES ($1, $2, $3, 'started')
       ON CONFLICT (account_id, idem_key) DO NOTHING
       RETURNING idem_key`,
      [req.accountId, key, hash]);

    if (claimed.rowCount === 0) {
      const { rows: [row] } = await tx.query(
        `SELECT request_hash, state, response_code, response_body
           FROM idempotency_keys
          WHERE account_id = $1 AND idem_key = $2
          FOR UPDATE`,
        [req.accountId, key]);
      if (!row.request_hash.equals(hash)) {
        return json(422, { error: 'Idempotency-Key reused with a different request' });
      }
      if (row.state === 'completed') return json(row.response_code, row.response_body);
      return json(409, { error: 'A request with this key is still in progress' });
    }

    const order = await insertOrder(tx, req.accountId, req.body);
    const body = { id: order.id, status: order.status };
    await tx.query(
      `UPDATE idempotency_keys
          SET state = 'completed', response_code = 201, response_body = $3
        WHERE account_id = $1 AND idem_key = $2`,
      [req.accountId, key, body]);
    return json(201, body);
  });
}

A few details carry more weight than they appear to. The fingerprint is computed over the operation and a canonical form of the body, with stable key order and normalised numbers, otherwise two serialisations of the same request look different and one key reused on another route goes unnoticed. If validation fails inside the transaction, the rollback removes the key as well, so a client that fixes its request can try again. And in PostgreSQL at the default READ COMMITTED isolation level, a concurrent duplicate's ON CONFLICT DO NOTHING claim waits for the first transaction to finish: if it commits, the duplicate finds a completed row and replays it; if it rolls back, the duplicate claims the key itself. Under REPEATABLE READ or SERIALIZABLE the duplicate gets a serialization failure instead, which the handler must catch and retry. In this single-transaction version the started state is never visible to anyone else.

When the side effect lives somewhere else

The single transaction stops working the moment the operation includes a call to another system: a payment provider, an SMS gateway, a partner API. You cannot hold a database transaction open across a network call of unknown duration, and you cannot roll back a card charge by rolling back your own transaction.

The pattern then becomes a small state machine with recovery points. Commit the claimed key in started state first. Make the external call, passing a downstream idempotency key derived deterministically from yours, for instance the account, your key and the step name. Then commit the result and mark the key completed. This is where the 409 branch and the locked_at column earn their place: a concurrent duplicate sees started and is told to wait, and a recovery job picks up keys stuck in started past a timeout.

Recovery is only safe because the downstream call was itself idempotent. The job can simply repeat it with the same downstream key, and the provider returns the original result rather than charging again. Without a downstream key, the recovery job has to ask the provider "did this happen?" and many APIs cannot answer that question reliably. If you integrate card payments, production patterns for Stripe payment integration covers how provider-side idempotency and webhooks fit together; the important point here is that your key and the provider's key must be linked, not generated independently on each attempt.

Side effects that are not calls but notifications, such as emails, pushes and events for other services, should not be sent inside the handler at all. Record them in the same transaction as the business change and send them afterwards from that record, so a rolled-back attempt never emails anyone and a replayed attempt never emails twice.

Idempotency without a key table

Not every operation needs the machinery above. Often the cheapest idempotency is designing the operation so the second execution is naturally a no-op.

Client-generated identifiers turn creation into an upsert. If the client chooses the order's UUID and the server writes it with ON CONFLICT (id) DO NOTHING, a repeat creates nothing, and the stored row is the response. State transitions can be made conditional: UPDATE invoices SET status = 'paid' WHERE id = $1 AND status = 'open' changes one row the first time and zero rows after that, and the handler can treat zero rows on an already-paid invoice as success. HTTP's own conditional requests, If-Match with an ETag, make updates safe against both retries and concurrent edits.

For queue consumers, the equivalent of the key table is a processed-messages table keyed by message ID, written in the same transaction as the consumer's effects. The broker's message ID is only useful if the producer sets it per business event; an ID assigned on each publish attempt makes every retry look new.

Increments are the classic trap. UPDATE balances SET amount = amount + 10 is not idempotent under any retry policy. Either record the individual credit with a unique identifier and derive the balance, or protect the increment with one of the mechanisms above.

The failure modes that cause double effects

The mistakes that produce double effects are few, and each one is easy to look for in review.

The key is generated inside the retry loop. Each attempt gets a fresh key, so the server sees independent requests. The key must be created where the user's intent is created, and persisted if the client may restart.

Retention is shorter than the retry horizon. An offline-capable app that replays a queued request after three days, against keys kept for one, will create duplicates that look perfectly valid.

The response is stored before the work is durable, or the work is durable before the key is. Either ordering leaves a window where a crash produces a key without an effect, or an effect without a key. The two must commit together, or the gap must be closed by a recovery path as described above.

Errors are cached indiscriminately. Replaying a stored 500 forever for a request that failed before any side effect occurred punishes a client for a transient fault. Decide explicitly: failures before the side effect started should release the key; outcomes after it started, success or business rejection, should be stored and replayed. Stripe's v1 API draws the line in the same place: it saves results, 500s included, only once endpoint execution has begun, and does not save validation failures or concurrent-request conflicts.

Keys are shared across operations. A client that uses one key for "create order" and then the same key for "add item" will receive confusing 422s at best. Including the route or operation name in the fingerprint, as the handler above does, makes this detectable.

Before you call an endpoint retry-safe

A short review list for any endpoint or consumer that clients are allowed to retry:

1. Every operation that creates or changes something has a defined behaviour on repeat: key table, client-generated ID, conditional update or processed-message record.
2. Keys are generated once per intent by the client, scoped per account, and retained longer than the longest client retry window.
3. The request fingerprint covers the canonical body and the operation; a mismatch returns an error rather than a replay.
4. Key and business effect commit atomically, or a recovery job resolves keys stuck in progress.
5. Every external call carries a downstream idempotency key derived from yours.
6. Notifications and events are sent from committed records, never directly from the handler.
7. A test sends the same request twice concurrently and twice sequentially and asserts exactly one effect.

That last item is the one teams skip, and it is the only one that proves the contract rather than describing it. Where retries, queues and payments meet in one platform, this is foundation work rather than polish, and it belongs in the first version; it is part of how we approach web platform development. Later issues of Under the Hood build on it.

References

• RFC 9110: HTTP Semantics
• The Idempotency-Key HTTP Header Field (IETF draft)
• Making retries safe with idempotent APIs — Amazon Builders' Library
• Idempotent requests — Stripe API reference