A client calls your payment API. The request reaches your server, the charge succeeds, and then the response is lost somewhere on the way back — a load balancer restart, a timeout, a flaky mobile network. From the client’s point of view, nothing happened. So it retries. Now you’ve charged the customer twice.

This is not an edge case. It’s the default behavior of every distributed system that uses at-least-once delivery, which is nearly all of them. TCP can retransmit, load balancers can retry failed upstream connections, message queues redeliver unacknowledged messages, and clients implement their own retry logic on timeouts. If your operations aren’t safe to run twice, you will eventually run them twice, and someone will notice — usually in a support ticket, not a test.

Idempotency is the property that makes retries safe: performing the same operation multiple times produces the same result as performing it once. Building it in deliberately, rather than hoping duplicates never happen, is one of the highest-leverage things you can do for a production API.

Why “Just Don’t Duplicate Requests” Doesn’t Work

The instinct is to prevent duplicates at the source — debounce the button, disable the submit form, add client-side guards. These help, but they don’t solve the underlying problem, because duplicates can be introduced anywhere along the chain:

  • The client times out and retries, but the original request actually succeeded server-side.
  • A proxy or API gateway retries a request that failed with a 502, unaware the handler already ran.
  • A message consumer crashes after processing a message but before acknowledging it, so the broker redelivers it.
  • A background job scheduler fires the same job twice due to a leader-election hiccup.

None of these are client bugs. They’re the normal cost of doing business over unreliable networks. The fix has to live on the server, in the operation itself.

Idempotency Keys

The standard pattern for write APIs (payments, order creation, anything with a side effect) is the idempotency key: the client generates a unique token per logical operation and sends it with every attempt, including retries.

POST /api/charges HTTP/1.1
Content-Type: application/json
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7

{
  "amount": 4200,
  "currency": "usd",
  "customerId": "cus_9f8a2b"
}

The server records the key alongside the result the first time it sees it. On a retry with the same key, it returns the stored result instead of re-executing the operation.

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  request_hash TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'in_progress', -- in_progress | completed | failed
  response_body JSONB,
  response_status INT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at TIMESTAMPTZ NOT NULL DEFAULT now() + interval '24 hours'
);

CREATE INDEX idx_idempotency_keys_expires_at ON idempotency_keys (expires_at);

The request_hash column matters: if a client reuses the same key for a different payload, that’s a client bug, and you want to reject it loudly rather than silently return a stale response for the wrong request.

import { createHash } from "node:crypto";

interface IdempotentResult<T> {
  status: number;
  body: T;
}

async function withIdempotency<T>(
  key: string,
  requestBody: unknown,
  operation: () => Promise<IdempotentResult<T>>
): Promise<IdempotentResult<T>> {
  const requestHash = createHash("sha256")
    .update(JSON.stringify(requestBody))
    .digest("hex");

  const existing = await db.query(
    `SELECT status, request_hash, response_body, response_status
     FROM idempotency_keys WHERE key = $1`,
    [key]
  );

  if (existing.rows.length > 0) {
    const row = existing.rows[0];
    if (row.request_hash !== requestHash) {
      throw new ConflictError("Idempotency key reused with a different request body");
    }
    if (row.status === "completed") {
      return { status: row.response_status, body: row.response_body };
    }
    if (row.status === "in_progress") {
      throw new ConflictError("Request with this key is already being processed");
    }
  }

  await db.query(
    `INSERT INTO idempotency_keys (key, request_hash, status)
     VALUES ($1, $2, 'in_progress')
     ON CONFLICT (key) DO NOTHING`,
    [key, requestHash]
  );

  const result = await operation();

  await db.query(
    `UPDATE idempotency_keys
     SET status = 'completed', response_body = $2, response_status = $3
     WHERE key = $1`,
    [key, result.body, result.status]
  );

  return result;
}

The expires_at column with a TTL sweep is important for two reasons: unbounded growth of the table, and giving clients a reasonable window to retry (24 hours is a common default; payment providers often use longer windows).

-- Run periodically (cron, pg_cron, or a scheduled job)
DELETE FROM idempotency_keys WHERE expires_at < now();

Natural Idempotency: Let HTTP Semantics Do the Work

Not every operation needs a client-generated key. HTTP methods carry idempotency semantics by design, and leaning on them removes a whole class of problems.

MethodIdempotent by specPractical meaning
GETYesSafe to retry freely, no side effects
PUTYes”Set this resource to this state” — retrying produces the same end state
DELETEYesDeleting an already-deleted resource is a no-op, not an error
POSTNoEach call is conventionally “create a new thing” — retries create duplicates
PATCHNo (usually)Depends on whether the patch is a delta or a full replacement

The practical takeaway: prefer PUT with a client-supplied identifier over POST when the operation is “create or update a specific resource.”

// Non-idempotent: every retry creates a new subscription
app.post("/api/subscriptions", async (req, res) => {
  const subscription = await db.subscriptions.create({
    userId: req.body.userId,
    planId: req.body.planId,
  });
  res.status(201).json(subscription);
});

// Idempotent: the client owns the resource identity, retries converge
app.put("/api/subscriptions/:id", async (req, res) => {
  const subscription = await db.subscriptions.upsert({
    where: { id: req.params.id },
    update: { planId: req.body.planId },
    create: { id: req.params.id, userId: req.body.userId, planId: req.body.planId },
  });
  res.status(200).json(subscription);
});

DELETE deserves the same treatment. Returning a 404 on the second delete attempt is technically accurate but operationally annoying — most clients treat “already gone” as success:

app.delete("/api/subscriptions/:id", async (req, res) => {
  await db.subscriptions.deleteMany({ where: { id: req.params.id } });
  // deleteMany doesn't throw if nothing matched — deletion is idempotent either way
  res.status(204).send();
});

Idempotency in Background Jobs and Queue Consumers

Queues almost universally guarantee at-least-once delivery, not exactly-once — the broker will redeliver a message if it doesn’t get an acknowledgment in time, even if your worker already finished processing it. If your job handler isn’t idempotent, every redelivery is a bug waiting to happen: double-charged invoices, duplicate emails, double-decremented inventory.

The same dedup-table pattern applies, keyed on a stable identifier from the message itself (not one you generate, since you need to recognize the same message across redeliveries):

CREATE TABLE processed_jobs (
  job_id TEXT PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
async function handleInventoryDecrement(message: QueueMessage) {
  const jobId = message.id; // stable across redeliveries, set by the producer

  const inserted = await db.query(
    `INSERT INTO processed_jobs (job_id) VALUES ($1)
     ON CONFLICT (job_id) DO NOTHING
     RETURNING job_id`,
    [jobId]
  );

  if (inserted.rows.length === 0) {
    // Already processed — acknowledge and move on, do not repeat the side effect
    return message.ack();
  }

  await db.query(
    `UPDATE inventory SET quantity = quantity - $1
     WHERE product_id = $2 AND quantity >= $1`,
    [message.body.quantity, message.body.productId]
  );

  await message.ack();
}

Two details matter here. First, the INSERT ... ON CONFLICT DO NOTHING ... RETURNING pattern does the “have I seen this before” check and the “claim it” step atomically — without it, two concurrent workers picking up a redelivered message could both pass a naive SELECT check before either writes. Second, the ack happens after the side effect, not before — acking early and then crashing before the update runs would lose the job entirely, trading duplicate-processing risk for message-loss risk, which is almost always the wrong trade.

For jobs whose side effect is itself naturally idempotent (see the next section), you can skip the dedup table entirely and save a database round-trip per message.

Making the Operation Itself Idempotent

The dedup table approach treats idempotency as bookkeeping bolted onto an operation. Often it’s cheaper to make the operation mathematically idempotent instead, so redelivery is a non-event rather than something you have to detect.

// Bad: Not idempotent — repeated execution keeps incrementing
async function applyLoyaltyPoints(userId: string, points: number) {
  await db.query(
    `UPDATE users SET loyalty_points = loyalty_points + $1 WHERE id = $2`,
    [points, userId]
  );
}

// Good: Idempotent — converges to a fixed target regardless of retries
async function setLoyaltyPointsForOrder(userId: string, orderId: string, points: number) {
  await db.query(
    `INSERT INTO loyalty_ledger (user_id, order_id, points)
     VALUES ($1, $2, $3)
     ON CONFLICT (order_id) DO NOTHING`,
    [userId, orderId, points]
  );
  // The user's total is a derived SUM over the ledger, never mutated directly
}

The second version reframes “add points” as “record the fact that this order earned these points,” keyed uniquely by order_id. Running it ten times has the same effect as running it once, because the ledger row can only be inserted once. The total balance becomes a read-time aggregation rather than a write-time accumulation — a small design shift with a large reliability payoff.

Where Idempotency Keys Come From

Whichever pattern you use, the key needs three properties: it must be generated by the party that can detect a retry (usually the client, since it’s the one deciding to resend), it must be stable across retries of the same logical operation, and it must be unique across different operations. A UUID v4 generated once per user action (not per HTTP attempt) and stored in the client until success is the common baseline; Stripe’s API and most modern payment gateways follow exactly this contract, and it’s worth mirroring their approach rather than inventing your own.

Key Takeaways

  • At-least-once delivery is the default in distributed systems — assume every operation can run more than once.
  • Idempotency keys let clients safely retry unsafe operations (POST) by having the server memoize the first result.
  • Store the request hash alongside the key so a reused key with a different payload is rejected, not silently misapplied.
  • Prefer naturally idempotent HTTP semantics (PUT with a client-owned ID, DELETE as a no-op) over bolting idempotency onto POST.
  • In queue consumers, dedup on the message’s own stable ID using an atomic INSERT ... ON CONFLICT, and ack only after the side effect completes.
  • Where possible, redesign the operation itself to be idempotent (ledger + derived total) instead of detecting and blocking duplicates.
  • Always TTL your idempotency/dedup tables — unbounded growth is a slow-motion outage.