I have a side project with a scheduled job that publishes one piece of content at a time to a CMS over its REST API. Two schedulers trigger it, a primary cron and a backup, so the very first thing the job does is ask the CMS a question: has anything been published in the last couple of hours? If yes, it exits cleanly. That check is the only thing standing between me and a double publish.

One morning the job logged its startup banner and then, about ten seconds later, died with fetch failed. No other output. It had not even reached the duplicate check’s answer. A single connection timeout to the CMS killed the whole run.

Fixing that is easy. Fixing it without creating a worse bug takes a little more thought, and the thought is what this post is about.

The tempting fix is the wrong one

The first instinct is to wrap the startup check in a try/catch and move on:

async function recentlyPublished() {
	try {
		const posts = await getJson(`${API}/posts?after=${twoHoursAgo()}`);
		return posts.length > 0;
	} catch {
		return false; // don't let a blip kill the run
	}
}

This turns a crash into a quiet success, and that is exactly the problem. If the CMS is down, recentlyPublished() now says “no”, which means “go ahead and publish”. The guard that exists to prevent duplicates now waves them through whenever the API is unreachable, which is precisely when you can least see what is going on.

A failed read is not an empty answer. “I could not find out” and “there is nothing there” are different results, and code that collapses them into one value will eventually act on the wrong one.

Three questions before you retry anything

The fix I shipped asks three questions about every request.

Is the request safe to repeat? A GET or HEAD does not change anything on the server, so sending it twice costs nothing. A POST that timed out might have reached the server and succeeded before the connection dropped. Retrying it blindly can create the second post you were trying to avoid. So reads get retries and writes get exactly one attempt. (If a write really needs retries, give it an idempotency key first, which is a different post.)

Is the failure the kind that goes away? Connection resets, DNS hiccups, timeouts, and gateway errors like 502, 503, and 504 are worth another try. A 400, 401, or 404 is the server telling you something true about your request, and asking again will get the same answer three times, slower.

What happens when the retries run out? The job has to fail, with an error that says which URL and why. It must never fall back to a default value.

The helper

Here is the shape of it, trimmed down:

const READ_TRIES = 3;

function triesFor(options = {}) {
	const method = String(options.method || 'GET').toUpperCase();
	return method === 'GET' || method === 'HEAD' ? READ_TRIES : 1;
}

function causeOf(err) {
	const c = err?.cause;
	if (!c) return err?.code ? String(err.code) : '';
	return [c.code, c.message].filter(Boolean).join(' ');
}

function isTransient(err) {
	if ([408, 429, 502, 503, 504].includes(err?.status)) return true;
	const text = `${err?.message ?? ''} ${err?.code ?? ''} ${causeOf(err)}`;
	return /fetch failed|ECONNRESET|ETIMEDOUT|EAI_AGAIN|ECONNREFUSED|UND_ERR|socket hang up/i.test(
		text,
	);
}

async function withRetry(fn, { tries, url, sleep, log = console.warn }) {
	let last;
	for (let attempt = 1; attempt <= tries; attempt++) {
		try {
			return await fn();
		} catch (err) {
			last = err;
			if (!isTransient(err) || attempt === tries) break;
			log(`attempt ${attempt}/${tries} failed for ${url}: ${err.message} ${causeOf(err)}`);
			await sleep(1000 * 2 ** (attempt - 1)); // 1s, then 2s
		}
	}
	if (!isTransient(last)) throw last;
	throw new Error(`unreachable after ${tries} tries: ${url} (${causeOf(last)})`, { cause: last });
}

export function request(
	url,
	options = {},
	{ sleep = (ms) => new Promise((r) => setTimeout(r, ms)) } = {},
) {
	return withRetry(
		async () => {
			const res = await fetch(url, options);
			if (!res.ok) throw Object.assign(new Error(`HTTP ${res.status}`), { status: res.status });
			return res.json();
		},
		{ tries: triesFor(options), url, sleep },
	);
}

A few details are doing more work than they look like.

The cause is where the real error lives. In Node, the built-in fetch throws a TypeError whose message is just fetch failed. The useful part, something like UND_ERR_CONNECT_TIMEOUT or ECONNRESET, is on err.cause. Logging only the message is how you end up with a run log that says “fetch failed” and nothing else, which is what I had. Pull the cause out and log it on every attempt.

The method decides the retry count, not the call site. Putting the rule in one place means nobody adding a new write later has to remember not to retry it. A POST simply gets one try.

A non-transient error passes through untouched. A 404 comes out as the original error on the first attempt, so the caller can still tell “not found” apart from “could not connect”.

sleep is injectable. That makes the retry path testable without real waits. A test can hand in a fake fetch that fails twice and then succeeds, plus a sleep that records the delays, and assert that the job got its answer, logged two warnings, and waited one and then two seconds. Another test makes every attempt fail and asserts that the error names the URL.

Short backoff on purpose

Three tries with one and two second waits is a deliberately small budget. This is a scheduled job with a backup trigger, not a user waiting on a page. If the API is down for more than a few seconds, the right move is to fail this run, show up red in the logs, and let the next scheduled run try again. Long retry loops in a cron job mostly delay the moment you find out something is broken, and they can push one run into the window of the next one.

If you run many clients against the same server, add jitter to the delay so they do not all come back at the same instant. For one job, a fixed doubling is fine.

The tradeoffs

Retries hide flakiness. A dependency that fails on the first attempt half the time will look healthy in your success metrics. That is why every retried attempt logs a warning: the run succeeds, but the evidence is still there for anyone who looks.

Failing loudly also means more red runs on days when the API is having a bad time. I would much rather have a red run than a duplicate publish, but that is a judgment about this job. For a read whose answer is truly optional, like a related links widget, treating failure as “show nothing” is reasonable. The test is whether the empty answer leads to an action. If the code does something because the list was empty, an outage must not be allowed to look like an empty list.

The rule I keep coming back to is short: retry what is safe to repeat, stop on errors that will not change, and when you run out of tries, say so instead of guessing.