Retries, backoff and circuit breakers

Retries fix blips. Backoff with jitter stops retries from becoming an attack on your own systems. A circuit breaker stops calling a dependency that is down.

How it works

Retry only what is safe to repeat (idempotent calls) and only errors that might go away: timeouts, 429, 5xx. Wait longer each time, doubling, with random jitter so a thousand clients don't retry at the same instant. Full jitter means wait a random time between 0 and base x 2^attempt, with a cap. A circuit breaker counts failures in a row. Past a threshold it opens and fails fast for a cool-down, which frees your threads and gives the dependency room to recover. Then one trial call goes through (half-open): success closes the breaker, failure opens it again.

Reach for it when

• Every call to another service or a third party API
• Workers calling flaky partners: SMS, payments, webhooks
• Anywhere a slow dependency could pile up threads or connections

Copy// Exponential backoff with full jitter: wait random(0, min(cap, base * 2^attempt)).
const backoff = (attempt, { base = 100, cap = 10000, rand = Math.random } = {}) =>
  Math.floor(rand() * Math.min(cap, base * 2 ** attempt));
// After `threshold` failures in a row: fail fast for `coolMs`, then let one trial call through.
class CircuitBreaker {
  constructor({ threshold = 3, coolMs = 5000, now = Date.now } = {}) {
    Object.assign(this, { threshold, coolMs, now, state: "closed", fails: 0, openedAt: 0 });
  }
  call(fn) {
    if (this.state === "open") {
      if (this.now() - this.openedAt < this.coolMs) throw new Error("circuit open"); // protect the callee
      this.state = "half-open"; // cool-down over: one trial
    }
    try {
      const r = fn();
      this.state = "closed";
      this.fails = 0;
      return r;
    } catch (err) {
      if (this.state === "half-open" || ++this.fails >= this.threshold) {
        this.state = "open";
        this.openedAt = this.now();
      }
      throw err;
    }
  }
}

Traps

• Retries at every layer multiply: 3 layers x 3 retries = 27 calls. Retry in one layer.
• Retrying non idempotent writes without an idempotency key.
• No timeouts: a call that hangs never fails, so nothing trips. Set timeouts first.
• Fixed delays without jitter sync clients into waves.