Split a slow request into pages, but a short page is not the last page

  • javascript
  • apis
  • reliability

I have a side project with a nightly job that scores the newest 200 pieces of content on a site. It asks the site’s API for all 200 in one request, computes a few numbers, and appends one row to a metrics file. Most nights that works. A few nights a month there is no row, and the log shows the same story every time: the CDN in front of the site gave up waiting on the origin, or the connection reset while the response was still being built.

The homepage was fast on those same nights, so the server was not down. It was this one request that was slow. The endpoint does extra work per item before it responds, so its response time grows with the page size, and 200 was the largest size anything asked it for. The fix I landed on is simple and boring, and it has one trap in it that is worth writing down.

The approach: same set, smaller requests

Instead of one request for 200 items, make four requests for 50 items each, newest first, and stitch them together. Each request gets its own timeout and one retry. The goal is that no single request is big enough to run past what the CDN will wait for.

The important constraint is that the result has to be the same set the single request returned. This job feeds a metric I compare day over day, so if paging quietly changed which items were scanned, the numbers would drift and I would blame the content instead of the code. Before trusting it, I fetched both ways back to back and compared the ordered id lists. They matched exactly.

Here is the shape of it:

async function fetchNewest(limit, { per = 50 } = {}) {
	per = Math.min(per, limit);
	const pages = Math.ceil(limit / per);
	const seen = new Set();
	const items = [];

	for (let page = 1; page <= pages; page++) {
		const started = Date.now();
		const body = await getJson(`/items?limit=${per}&page=${page}&order=desc`);
		const batch = body.data ?? [];
		console.error(`page ${page}: ${batch.length} items in ${Date.now() - started} ms`);

		for (const item of batch) {
			if (!seen.has(item.id)) {
				seen.add(item.id);
				items.push(item);
			}
		}

		const total = body.pagination?.total;
		const shortPage = batch.length < per;
		const nothingLeft = total == null || page * per >= total;
		if (shortPage && nothingLeft) break;
	}

	return items.slice(0, limit);
}

A few details in there earn their keep.

Dedupe by id, keep the first one. If something new is published between page 1 and page 2, everything shifts down by one and the last item of page 1 shows up again at the top of page 2. The Set handles that. The opposite case, an item skipped across a page boundary, can also happen. I accepted that, because new content arrives a few times a day and the job takes seconds. If your data changes faster, page by a cursor (for example, “older than this id”) instead of by page number.

Truncate to the limit. Dedupe can leave you short and a generous last page can leave you long. Slicing at the end means the caller always gets the size it asked for when the data is there.

Log the timing per page. This was the most useful line in the whole change. On a cold server, the first two pages took over 20 seconds each for just 50 items. A few minutes later every page came back in a few seconds, and so did the single big request. That told me the failures were about a cold origin, and that a cold 200 item request could easily run past a 60 second limit. It does not prove the theory, but it makes it a lot more believable, and it costs nothing.

The trap: a short page does not always mean you are done

The usual way to stop a paging loop is “if a page comes back with fewer items than I asked for, that was the last page.” I almost wrote exactly that.

The problem is that this endpoint filters after it queries. It asks the database for 50 rows, then drops any row that is missing some related data, and returns what is left. So a page can come back with 48 items while there are thousands more behind it. With the naive rule, a single incomplete row on page 1 would end a 200 item scan at about 50, and the metric would silently be computed over a quarter of the usual data.

So the loop only treats a short page as the end when the server agrees. If the response includes a total and page * per has reached it, we really are done. If there is no total to check, a short page is the only signal there is, so it stops there too. When the total says there is more, it keeps going.

The general lesson is that “short page means last page” is an assumption about the server, not a law of paging. It holds when the server returns exactly the rows its query matched. It breaks the moment anything removes rows after the limit is applied: permission filtering, soft deletes, post-processing, or a join that can come up empty. If you do not own the endpoint, check for this before you rely on it.

Fail the whole thing, not part of it

Each page gets one retry on a network error, a timeout, or a 5xx. A 4xx is never retried, so a bad credential still fails in a fraction of a second. If a page still fails after its retry, the whole scan fails and no row is written.

It is tempting to save what you have. Three pages out of four is 150 items, which looks like most of the answer. But this job writes a number that gets compared against yesterday’s number, and 150 items is a different denominator. A missing row is obvious when I look at the history. A row computed over the wrong set looks completely normal and is wrong. When partial data would be mistaken for complete data, a gap is the honest result.

One more detail on the timeout: put it on the whole request, body included, not just the time until headers arrive. A server that sends headers quickly and then stalls the body is exactly the slow, expensive case you are trying to catch. In Node, passing AbortSignal.timeout(60_000) as the fetch signal covers both.

The tradeoffs

Four requests cost more total time than one on a warm server, and they put four hits in your logs instead of one. For a nightly job that does not matter. For something on a request path, it might.

Paging by page number is only as stable as the sort order. Sort by something that does not change, like publish time, and break ties by id if you can. Otherwise items can swap places between pages.

And this is a mitigation, not a cure. If the origin is slow because it is cold, smaller requests make each one more likely to finish, but they do not make the server faster. The honest way to judge the change is to measure it: count how many days in the next two weeks have a row, and compare that to the two weeks before. If small pages also fail on days when everything else is fast, then request size was never the problem, and I will go look at the server instead.