Put a timeout on every child process your script spawns
Most glue scripts I write follow the same shape. Read some state, loop over a list of items, shell out to a CLI for each one, and write the result back. It’s the fastest way to connect tools that were never meant to talk to each other, and it works great right up until one of those child processes simply never returns.
That happened to me on a side project that runs a small automation loop on a schedule. One step reconciles pull request status into a backlog file: for every item marked “in review,” it asks the GitHub CLI whether the PR has merged or closed, then updates the item. A merge is what starts the clock on measuring whether a change worked, and no person is around at that moment, so this step is the only thing that moves items forward.
One morning the gh pr view call hung for over ten minutes. Then it did it again. The script
had no timeout, so the whole reconcile sat there, and nothing after it in the cycle ran either.
Nothing crashed and nothing logged an error. Items just quietly stopped advancing.
A missing timeout is a decision you didn’t make
When you call execFileSync (or subprocess.run, or backticks in a shell script) without a
timeout, you’ve decided that you’re willing to wait forever. You almost never mean that. You
mean “this should take a second or two, and if it takes much longer, something is wrong.”
So the first fix is to say that out loud, for every child, not just the one that hung this time:
import { execFileSync } from 'node:child_process';
const GH_TIMEOUT_MS = Number(process.env.GH_TIMEOUT_MS) || 20000;
const WRITE_TIMEOUT_MS = Number(process.env.WRITE_TIMEOUT_MS) || 30000;
const run = (args, opts = {}) =>
execFileSync(args[0], args.slice(1), {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
...opts,
}).trim();
const gh = (args) =>
run(args, {
timeout: GH_TIMEOUT_MS,
env: { ...process.env, GH_PROMPT_DISABLED: '1', GH_NO_UPDATE_NOTIFIER: '1' },
});
A few details in there matter more than they look:
stdinisignore. If a child ever decides to read from standard input, it inherits your script’s stdin by default. In a scheduled job that might be a pipe nobody writes to, and the child waits on it forever. Closing it means the child gets end of file immediately.- Interactive prompts are turned off. Many CLIs will happily stop and ask a question when they think a human is present. Most have an environment variable or flag to disable that. Use it in automation, always.
- Background update checks are turned off. Some CLIs fork a little process to check for a new version. If that process holds your captured stdout pipe open, your call can’t finish even though the real work is done.
- The timeouts are configurable. A sensible default lives in the code, and an environment variable can stretch it on a slow network without a code change.
Fail one item, not the whole run
A timeout turns “hang forever” into “throw an error,” which is only half the fix. If that
error kills the loop, one bad item still blocks every good one behind it. So each item gets its
own try, the failure is logged in one line, and the loop moves on.
It’s also worth telling a timeout apart from an ordinary error, because they lead you to very
different places when you debug. Node reports a sync timeout as ETIMEDOUT on some platforms
and only as a SIGTERM kill with a null exit status on others, so I check for both:
function describeError(e, timeoutMs) {
if (e.code === 'ETIMEDOUT' || (e.status === null && e.signal === 'SIGTERM')) {
return `timed out after ${timeoutMs}ms`;
}
const stderr = String(e.stderr || '')
.trim()
.split('\n')[0];
return stderr || String(e.message).split('\n')[0];
}
At the end, the script prints a single summary like reconciled 3, unchanged 5, failed 1.
It still exits 0, because a partial reconcile shouldn’t abort the rest of the cycle, but anyone
reading the log can see the gap at a glance.
Make the dangerous failure loud
Not all failures are equal. If the read fails, nothing is lost: the item stays in review and the next run tries again. The nasty case is when the read succeeds, you learn the PR merged, and then the write that records it fails. Next time the item might look the same, or the information might be gone, and the system never finds out.
So that case gets its own message. Instead of a generic error, the script says the transition was observed but not recorded, and prints the exact command a person could paste to replay it:
try {
run(['node', backlogCli, 'set', item.id, `status=${status}`], { timeout: WRITE_TIMEOUT_MS });
reconciled++;
} catch (e) {
console.log(`${item.id}: LOST TRANSITION, ${item.pr} is ${pr.state} but the write failed`);
console.log(`${item.id}: replay with: node backlog.mjs set ${item.id} status=${status}`);
failed++;
}
The general rule: when you’ve learned something you can’t easily learn again, and you failed to save it, that’s the error worth shouting about.
Then go find the actual cause
Timeouts keep the run alive, but they’re a seatbelt, not a diagnosis. Once the script stopped hanging silently, I could look at why the call was slow in the first place.
The answer was the working directory. Every child was being launched with cwd set to a repo
checkout owned by a different user. The CLI probes the git repository in its current directory
even when you hand it a full PR URL, and when that directory isn’t readable by the process
running the loop, the probe just never comes back. Neither child needed a repo directory at
all: the CLI can resolve everything from the URL, and the backlog tool uses absolute paths. So
the cwd option went away, and the hang went with it.
That’s a pattern I see a lot. Options like cwd and env get copied into a shared helper
“just in case,” and then every call inherits assumptions that only one of them needed. When a
child process misbehaves, the context you launched it with is the first thing to question.
The tradeoffs
Picking timeout values is guesswork at first. Too short and you get false failures on a slow day; too long and a stuck run still wastes minutes. I start around ten times the normal duration and adjust after watching real logs for a week.
A timeout also doesn’t undo anything. If a write times out halfway, you need the operation
itself to be safe to retry, or at least easy to check. And execFileSync blocks the event
loop, which is fine for a sequential cron script but a bad fit for anything serving requests;
there, use the async execFile with an AbortSignal instead.
None of this is clever. It’s just refusing to let “wait forever” be the default, and making sure the one failure that loses information can’t happen quietly.