A downstream service slows down for ten seconds. Your service has a simple retry loop: if a call fails, try again, up to three times. During those ten seconds, every request now generates up to four calls instead of one. The struggling service receives four times the load exactly when it can least afford it, falls over completely, and what was a hiccup becomes an outage that lasts until someone restarts things by hand.

Retries are one of the most useful reliability tools and one of the most reliable ways to cause an incident. The difference between the two outcomes is a handful of rules: wait between attempts, spread the waiting randomly, cap the total effort, and only retry what is safe to retry. This article builds up a correct retry implementation step by step.

Why Retry at All

Many failures are transient: a dropped connection, a brief leader election, a 503 during a deploy, a rate limit that resets in a second. Retrying turns these from user-visible errors into invisible delays. The key word is transient. Retrying a validation error or a missing resource only wastes time, because the answer will not change.

So the first decision is classification. Before writing any loop, decide which errors are retryable.

const RETRYABLE_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]);

export function isRetryable(error: unknown): boolean {
  if (error instanceof HttpError) return RETRYABLE_STATUS.has(error.status);
  if (error instanceof Error) {
    // network level failures surfaced by fetch / undici / node
    const code = (error as NodeJS.ErrnoException).code;
    return ["ECONNRESET", "ETIMEDOUT", "ECONNREFUSED", "EAI_AGAIN"].includes(code ?? "");
  }
  return false;
}

export class HttpError extends Error {
  constructor(public readonly status: number, message?: string) {
    super(message ?? `HTTP ${status}`);
  }
}

Client errors such as 400, 401, 403, 404 and 422 are deliberately absent. A 429 is included because it means “try later”, ideally after the delay the server specified.

Step One: Do Not Retry Immediately

The worst retry loop has no delay:

// Do not do this
for (let attempt = 0; attempt < 3; attempt++) {
  try {
    return await call();
  } catch {}
}

If the failure was a service being overloaded, an immediate retry lands on it within milliseconds. A fixed delay of one second is better, but if a thousand clients failed together, a thousand clients will retry together one second later, and again one second after that. Fixed delays preserve synchronization.

Step Two: Exponential Backoff

Exponential backoff doubles the wait after each failure: 100 ms, 200 ms, 400 ms, 800 ms, and so on, up to a maximum. Early retries are quick for blips that clear instantly; later retries back off sharply to give a genuinely unhealthy service room to recover.

function backoffDelay(attempt: number, baseMs = 100, capMs = 10_000): number {
  return Math.min(capMs, baseMs * 2 ** attempt);
}

The cap matters. Without it, attempt 10 would wait over a minute and a half with a 100 ms base, which is rarely what you want in a request path.

Step Three: Add Jitter

Backoff alone still synchronizes clients. If a thousand clients all failed at the same moment, they all wait exactly 200 ms, then all retry together, then all wait exactly 400 ms. The load arrives in coordinated waves. Jitter adds randomness so retries spread out over time.

Three common variants:

// Full jitter: random between 0 and the exponential value
function fullJitter(attempt: number, baseMs = 100, capMs = 10_000): number {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.random() * exp;
}

// Equal jitter: half fixed, half random
function equalJitter(attempt: number, baseMs = 100, capMs = 10_000): number {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return exp / 2 + Math.random() * (exp / 2);
}

// Decorrelated jitter: next delay depends on the previous one
function decorrelatedJitter(prev: number, baseMs = 100, capMs = 10_000): number {
  return Math.min(capMs, baseMs + Math.random() * (prev * 3 - baseMs));
}

Full jitter is a good default. It spreads load the most evenly, at the cost that an individual retry can occasionally fire almost immediately. Equal jitter guarantees a minimum wait. Decorrelated jitter avoids tracking the attempt number and tends to spread clients well. All three are dramatically better than none; do not agonize over the choice.

A Complete Retry Helper

Putting classification, backoff and jitter together, with a cancellation signal and respect for Retry-After:

export interface RetryOptions {
  maxAttempts?: number;        // total tries including the first
  baseMs?: number;
  capMs?: number;
  signal?: AbortSignal;
  shouldRetry?: (error: unknown) => boolean;
}

const sleep = (ms: number, signal?: AbortSignal) =>
  new Promise<void>((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal?.addEventListener("abort", () => {
      clearTimeout(timer);
      reject(signal.reason);
    }, { once: true });
  });

export async function retry<T>(
  operation: (attempt: number) => Promise<T>,
  { maxAttempts = 4, baseMs = 100, capMs = 5_000, signal, shouldRetry = isRetryable }: RetryOptions = {},
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    signal?.throwIfAborted();
    try {
      return await operation(attempt);
    } catch (error) {
      lastError = error;
      const isLast = attempt === maxAttempts - 1;
      if (isLast || !shouldRetry(error)) break;

      const delay = fullJitter(attempt, baseMs, capMs);
      await sleep(delay, signal);
    }
  }
  throw lastError;
}

// Usage
const user = await retry(() => fetchJson("https://api.example.com/users/42"), {
  signal: AbortSignal.timeout(15_000),
});

The AbortSignal.timeout gives the whole operation a deadline, so retries cannot extend a request beyond what the caller is willing to wait. See Structured Concurrency and Cancellation for more on propagating cancellation.

If the server sends a Retry-After header, honor it instead of your own calculation. The server knows its recovery state better than your formula does.

The same logic in Python, as a decorator:

import random
import time
from functools import wraps

def retry(max_attempts=4, base=0.1, cap=5.0, retry_on=(ConnectionError, TimeoutError)):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return fn(*args, **kwargs)
                except retry_on:
                    if attempt == max_attempts - 1:
                        raise
                    time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))
        return wrapper
    return decorator

@retry()
def fetch_report(report_id: str) -> dict:
    ...

If you prefer composition over a free function, the Decorator pattern shows how to wrap a client with retry behavior without touching its callers.

Step Four: Only Retry Safe Operations

A retry is a duplicate request from the server’s point of view. If the first attempt actually succeeded but the response was lost, retrying a POST /payments may charge the customer twice. Reads and naturally idempotent writes (PUT, DELETE) are safe to retry. Non-idempotent writes are only safe if you attach an idempotency key that lets the server recognize and deduplicate the repeat. That design is covered in depth in Idempotency in APIs and Background Jobs. A good rule: no retry without idempotency, and no idempotency without a plan for what a duplicate does.

Step Five: Retry Budgets

Per-request limits are not enough. If every request may retry three times, then during a full outage the load multiplier is four, no matter how nicely each request backs off. A retry budget caps retries as a fraction of overall traffic: for instance, retries may add at most 10 percent on top of the original request volume.

export class RetryBudget {
  private requests = 0;
  private retries = 0;
  private windowStart = Date.now();

  constructor(private readonly ratio = 0.1, private readonly windowMs = 10_000, private readonly minRetries = 3) {}

  private roll(now: number) {
    if (now - this.windowStart > this.windowMs) {
      this.requests = 0;
      this.retries = 0;
      this.windowStart = now;
    }
  }

  recordRequest(now = Date.now()) {
    this.roll(now);
    this.requests += 1;
  }

  tryAcquireRetry(now = Date.now()): boolean {
    this.roll(now);
    const allowed = Math.max(this.minRetries, Math.floor(this.requests * this.ratio));
    if (this.retries >= allowed) return false;
    this.retries += 1;
    return true;
  }
}

Wire it into the retry helper: call recordRequest() on every first attempt and check tryAcquireRetry() before sleeping. When the budget is exhausted, fail fast. The result is that a sick dependency sees at most a modest increase in traffic from you rather than a multiplication.

Circuit breakers complement budgets. After enough consecutive failures the breaker opens and calls fail immediately for a cooldown period, which stops you from spending time and sockets on a service that is clearly down.

Retries Compound Across Layers

Suppose service A calls B, which calls C, and each layer retries three times. A single failure at C can produce 3 x 3 x 3 = 27 attempts. This is called retry amplification, and it is a frequent cause of cascading failures. Mitigations:

  • Retry at one layer only, typically the one closest to the user or the edge, and let inner layers fail fast.
  • Propagate deadlines so inner calls know how much time remains and do not retry past it.
  • Use retry budgets so the amplification is bounded even if multiple layers retry.

Testing Retry Logic

Retry code is easy to get subtly wrong and rarely exercised in production until it matters. Inject the clock and the random source so tests run instantly and deterministically.

import { describe, it, expect, vi } from "vitest";

describe("retry", () => {
  it("retries transient errors and then succeeds", async () => {
    const op = vi.fn()
      .mockRejectedValueOnce(new HttpError(503))
      .mockResolvedValueOnce("ok");
    const result = await retry(op, { baseMs: 1, capMs: 2 });
    expect(result).toBe("ok");
    expect(op).toHaveBeenCalledTimes(2);
  });

  it("does not retry client errors", async () => {
    const op = vi.fn().mockRejectedValue(new HttpError(404));
    await expect(retry(op, { baseMs: 1 })).rejects.toThrow("HTTP 404");
    expect(op).toHaveBeenCalledTimes(1);
  });
});

Observability for Retries

Retries hide failures from users, which is both their purpose and their danger. A dependency that fails 30 percent of the time and succeeds on the second attempt looks healthy from the outside while quietly doubling its own load. Emit a metric or structured log line for every retry, labelled with the dependency, the error class and the attempt number. Alert on the retry rate, not only on final failures. A rising retry rate is usually the earliest warning that something downstream is degrading, long before users notice errors.

It also helps to record the final outcome separately: succeeded on first try, succeeded after retry, and exhausted. The ratio between these tells you whether your attempt limit and backoff parameters are tuned sensibly. If almost nothing ever succeeds on the third attempt, the third attempt is just extra load and the limit can come down.

Common Mistakes

  • Retrying every exception, including programming errors and validation failures.
  • Using a fixed delay or backoff without jitter, which keeps clients synchronized.
  • Retrying non-idempotent writes with no idempotency key.
  • No overall deadline, so a request retries long after the user gave up.
  • Stacking retries at every layer of a call chain.
  • Swallowing the final error. After the last attempt, rethrow the original failure with its context intact.
  • Logging nothing. Record each retry with attempt number and reason, or you will not know a dependency is flaky until it breaks.

Choosing the Parameters

Defaults should come from the dependency, not from habit. Set the base delay near the typical time a transient fault takes to clear, often tens to hundreds of milliseconds for a network blip and seconds for a failover. Set the cap below the caller’s own timeout, and the attempt count low: three or four total tries is usually plenty. More attempts rarely rescue a request that failed three times; they mostly prolong the damage. Revisit the numbers whenever you change timeouts elsewhere, because retry settings and timeouts form one system.

When Not to Retry

Do not retry when the error is permanent, when the operation is not idempotent and you cannot make it so, when the caller has already timed out, or when the downstream system explicitly signals overload and you are already past your budget. In interactive paths, a fast clear failure is often better than a slow uncertain success.

Key Takeaways

  • Classify errors first; only transient failures deserve a retry.
  • Use exponential backoff with a cap, and always add jitter to desynchronize clients.
  • Make retried operations idempotent, using idempotency keys for writes.
  • Bound total effort with deadlines and retry budgets, not just per-call attempt counts.
  • Retry at one layer of a call chain to avoid multiplicative amplification.
  • Test the logic with injected time and randomness, and log every retry.