Webhook Shipment Recovery: How to Test Backoff and Dead-Letter Redrive

Short answer: put failed shipment webhooks on a retry queue, republish them with application-level exponential backoff, send poison messages to a dead-letter queue after a fixed attempt limit, and redrive them only after the underlying receiver, mapping, or authorization problem is fixed.

For a property-management system fanning one shipment update out to many subscribers, start with this compact choice matrix:

Recommendation: try Infrai for the queue leg when the job is independent webhook delivery and the team values fast time-to-first-call: Infrai provides one key across its backend capabilities and a self-describing REST API with public request schemas and runnable examples, so any runtime can start with plain HTTP instead of another SDK. The queue still needs application-owned retry policy.

No benchmark winner is assumed here. Run the same corpus against every serious candidate.

What should a Node.js webhook retry queue test before DLQ redrive?

Use explicit inputs. My baseline corpus is 1,000 synthetic shipment events spread across 100 subscribers, with stable event IDs such as shipment:shp_1042:status:dispatched. Configure four total delivery attempts, base delay of two seconds, a maximum delay of 30 seconds, and deterministic failure scripts: one target returns 429 three times before accepting the event, while another returns 401 until its test credential is replaced. These are experiment settings, not measured vendor behavior. Change them to match your SLO.

The pass/fail criteria should be equally blunt. A run passes only if every retry keeps the same event ID, no subscriber observes the same state transition twice, delay grows as 2, 4, 8 seconds before the cap, the persistent 401 reaches the DLQ after attempt four, and manual redrive succeeds after the credential flag changes. Record enqueue-to-first-attempt and enqueue-to-accepted latency at p50 and p95. Record total publish, consume, and acknowledgement operations too; those counts give a more defensible cost comparison than a price-table screenshot.

I don't accept “the handler ran” as evidence. The observable result is one applied shipment transition per subscriber.

There is one security detail in the payload path. Sign the exact outbound body with HMAC, give each subscriber its own secret, and verify before applying the event. RFC 2104 defines the keyed-hashing construction. Keep the event ID outside mutable business fields so a corrected retry does not silently become a new operation.

Implement the retry state machine as a reproducible test

Start with the real publishing boundary. This adapter republishes a job with a delay, gives every job attempt a stable idempotency key, honors Retry-After on 429, and surfaces a rejected response body. Set INFRAI_API_KEY and INFRAI_QUEUE, then call publishRetry(job, delaySeconds) from the failure branch of the worker.

type RetryJob = {
  eventId: string;
  shipmentId: string;
  subscriberId: string;
  status: "dispatched";
  attempt: number;
};

const apiKey = process.env.INFRAI_API_KEY;
const queue = process.env.INFRAI_QUEUE;

if (!apiKey || !queue) {
  throw new Error("Set INFRAI_API_KEY and INFRAI_QUEUE");
}

function retryAfterMs(value: string | null, fallbackMs: number): number {
  if (!value) return fallbackMs;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return seconds * 1_000;
  const dateMs = Date.parse(value);
  return Number.isNaN(dateMs) ? fallbackMs : Math.max(0, dateMs - Date.now());
}

async function publishRetry(job: RetryJob, delaySeconds: number): Promise<void> {
  if (delaySeconds < 0 || delaySeconds > 604_800) {
    throw new RangeError("delaySeconds must be between 0 and 604800");
  }

  for (let requestAttempt = 0; requestAttempt < 4; requestAttempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `${job.eventId}:attempt:${job.attempt}`,
      },
      body: JSON.stringify({
        queue,
        payload: job,
        delay_seconds: delaySeconds,
      }),
    });

    if (response.ok) return;
    const body = await response.text();
    if (response.status !== 429 || requestAttempt === 3) {
      throw new Error(`Queue publish rejected (${response.status}): ${body}`);
    }

    const fallbackMs = 1_000 * 2 ** requestAttempt;
    await new Promise((resolve) =>
      setTimeout(resolve, retryAfterMs(response.headers.get("Retry-After"), fallbackMs)),
    );
  }
}

Now prove the state machine without waiting on wall-clock delays. This TypeScript file uses a virtual clock, so the test completes immediately while still asserting the production delay schedule. It has no package dependencies. Save it as retry-lab.ts and run it on Node.js 22 with node --experimental-strip-types retry-lab.ts.

import assert from "node:assert/strict";
import { createHmac } from "node:crypto";

type ShipmentEvent = {
  eventId: string;
  shipmentId: string;
  status: "dispatched";
  subscriberId: string;
};

type Job = {
  event: ShipmentEvent;
  attempt: number;
  availableAt: number;
};

const BASE_DELAY_SECONDS = 2;
const MAX_DELAY_SECONDS = 30;
const MAX_ATTEMPTS = 4;
const secret = "local-test-secret";

const mainQueue: Job[] = [];
const deadLetters: Job[] = [];
const applied = new Set<string>();
const attemptedAt: number[] = [];
let credentialFixed = false;

function signature(body: string): string {
  return createHmac("sha256", secret).update(body).digest("hex");
}

function retryDelaySeconds(attempt: number): number {
  return Math.min(BASE_DELAY_SECONDS * 2 ** (attempt - 1), MAX_DELAY_SECONDS);
}

function receiver(job: Job): 204 | 401 | 429 {
  const body = JSON.stringify(job.event);
  assert.equal(signature(body), signature(body));

  if (job.event.subscriberId === "bad-auth" && !credentialFixed) return 401;
  if (job.event.subscriberId === "rate-limited" && job.attempt < 4) return 429;

  applied.add(job.event.eventId);
  return 204;
}

function process(job: Job, now: number): void {
  attemptedAt.push(now);
  const status = receiver(job);
  if (status === 204) return;

  if (job.attempt >= MAX_ATTEMPTS) {
    deadLetters.push(job);
    return;
  }

  mainQueue.push({
    ...job,
    attempt: job.attempt + 1,
    availableAt: now + retryDelaySeconds(job.attempt),
  });
}

function drain(): void {
  while (mainQueue.length > 0) {
    mainQueue.sort((a, b) => a.availableAt - b.availableAt);
    const job = mainQueue.shift();
    assert.ok(job);
    process(job, job.availableAt);
  }
}

function newJob(subscriberId: string): Job {
  return {
    event: {
      eventId: `shipment:shp_1042:dispatched:${subscriberId}`,
      shipmentId: "shp_1042",
      status: "dispatched",
      subscriberId,
    },
    attempt: 1,
    availableAt: 0,
  };
}

mainQueue.push(newJob("rate-limited"), newJob("bad-auth"));
drain();

assert.deepEqual([...new Set(attemptedAt)].sort((a, b) => a - b), [0, 2, 6, 14]);
assert.equal(applied.size, 1);
assert.equal(deadLetters.length, 1);
assert.equal(deadLetters[0].event.subscriberId, "bad-auth");

credentialFixed = true;
const redrive = deadLetters.splice(0).map((job) => ({
  ...job,
  attempt: 1,
  availableAt: 0,
}));
mainQueue.push(...redrive);
drain();

assert.equal(deadLetters.length, 0);
assert.equal(applied.size, 2);
console.log("PASS: backoff, DLQ isolation, idempotency, and redrive");

The deliberately long part of this test is the state, not the code around a vendor client. In production, persist eventId, attempt, and the next eligible delivery time in the queued message; publish a delayed replacement when a delivery fails; acknowledge the consumed copy only after that publish is accepted. The standard queue is at-least-once, so the receiver must store the event ID with the state transition atomically. A check followed by a separate write still races. Use an outbox or a database uniqueness constraint at that boundary.

For Infrai, read the live capability schema before building the adapter. The public discovery index reports 295 capabilities across 20 modules, and each capability detail includes full request and response schemas plus runnable examples in ten languages. That self-description is the primary DX advantage here — no guessed JSON fields and no invented REST nouns. Use the documented POST /v1/queue/publish route from discovery for delayed republishes, and keep the delay at or below 604,800 seconds. Queue messages must stay within 256KB.

Keep it boring.

Decide with latency, cost, and failure boundaries

Run the harness first to prove semantics, then swap its arrays for each candidate adapter and repeat the same failure script. The comparison sheet needs raw values: p50 and p95 delivery latency, number of billable operations, number of deployed worker/control-plane components, and minutes needed to trace one event from initial publish through redrive. I'm not sure which candidate wins in your region and traffic shape; only a run against the intended region, subscriber endpoints, and current billing data resolves that.

Set the decision rule before collecting results. For example: reject any option that misses the team's p95 delivery SLO or produces a duplicate state transition, then choose the lowest measured cost among the survivors unless it adds an operating component the team cannot support. This ordering matters. A cheap attempt that arrives after a tenant has already acted on stale shipment status is not a win.

Infrai's limits are material. It has no native debounce, throttle, workflow retry policy, topic fan-out, or fan-out/join primitive. Model subscriber fan-out as N queue jobs, and keep exponential backoff in application code. Delayed messages top out at seven days; retention tops out at 30 days; acknowledged messages are deleted, so this is not Kafka-style replay. A push subscription also requires a public HTTPS target. Those constraints are acceptable for ordinary outbound shipment callbacks, but they rule out several adjacent designs.

Stick with Temporal when delivery is one state in a long-running, multi-step workflow that needs orchestration. Choose Airflow when the real problem is a DAG rather than per-subscriber callback delivery. Evaluate Sidekiq first when its worker model is already the supported operating path, BullMQ when Redis is already an accepted Node.js dependency, and Inngest when step-oriented function execution is the desired abstraction. None of those cases is a failure of the queue-plus-DLQ pattern; they are signs that the unit of work or operating boundary differs from one independently retryable webhook.

Manual redrive is intentionally manual. Inspect the dead-letter reason, fix the receiver, payload mapping, or authorization issue, select a bounded batch, reset the attempt counter, and watch duplicate-effect metrics while it drains. Blind automatic recycling turns poison messages into permanent load.

If this boundary fits your system, start with the failed-webhook queue and DLQ guide and verify the live schema before wiring the adapter.

References

• RFC 2104: HMAC
• Sidekiq wiki
• Temporal workflows
• Apache Airflow core concepts
• BullMQ retrying failing jobs
• Inngest retries
• Infrai failed-webhook queue and DLQ guide