API timeouts and missed webhooks happen when backend handlers wait synchronously on external third-party services that experience latency spikes. Wrapping external calls in explicit AbortController timeouts, implementing exponential backoff retries, and designing idempotent webhook ingestion ensures systems never drop critical transactions.
Webhook providers expect an immediate HTTP 200 response within 5 to 10 seconds. Doing heavy image processing, multiple database writes, or sending emails inside the initial handler triggers timeout retries.
When a network blip occurs after payment capture but before returning a response, clients retry the POST request. Without idempotency guards, duplicate charges or double database records occur.
By default, Node.js 'fetch' has no default timeout. If an upstream service stalls, the node thread waits indefinitely, leading to socket exhaustion.
export async function fetchWithRetry(
url: string,
options: RequestInit = {},
retries = 3,
backoffMs = 1000
): Promise<Response> {
const timeoutMs = 8000;
for (let attempt = 1; attempt <= retries; attempt++) {
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch(url, {
...options,
signal: controller.signal,
});
clearTimeout(timeoutId);
// Do not retry client 4xx errors; retry server 5xx errors
if (response.ok || (response.status >= 400 && response.status < 500)) {
return response;
}
} catch (err: any) {
clearTimeout(timeoutId);
if (attempt === retries) throw err;
}
// Exponential backoff delay with jitter
const delay = backoffMs * Math.pow(2, attempt - 1) + Math.random() * 200;
await new Promise((resolve) => setTimeout(resolve, delay));
}
throw new Error(`Failed to fetch ${url} after ${retries} attempts`);
}| Failure Mode | Root Cause | Resilience Pattern |
|---|---|---|
| Webhook 504 timeout | Synchronous heavy task in route | Return 200 immediately and queue background task |
| Duplicate payment charge | Retried without idempotency | Store unique Idempotency-Key in database/Redis |
| API hanging on third party | No explicit fetch timeout | Wrap request with AbortController (e.g. 8s timeout) |
| Upstream service rate limit (429) | Burst traffic without backoff | Implement exponential backoff with random jitter |
Enforce strict timeouts on every outgoing third-party request.
Record event_id in database before processing to prevent duplicate handling.
Offload long-running operations to background workers or BullMQ queues.
Production Verification
A production platform handling asynchronous contractor milestone payments and idempotent webhook processing.
An idempotency key is a unique token (like a UUID) sent with a request. If the server receives the same key twice within 24 hours, it returns the cached result without executing the charge a second time.
User-facing web requests should abort after 5 to 8 seconds. Background sync jobs can wait up to 30 seconds.
Jitter introduces slight random variations to retry delays. This prevents hundreds of retrying clients from hitting the recovering server simultaneously (the 'thundering herd' problem).
If this error is blocking your release build or production pipeline, Syed Alijah Muhammad can diagnose and deploy a clean fix. Send the repository access, error trace, and platform details.