When two fields disagree, repair the one that is actually wrong

  • ai
  • llm
  • testing

One of my side projects generates short articles from source material, in several versions for different readers. Each version gets a headline and a body, and before anything publishes, a set of checks in code decides whether the story is good enough. One of those checks compares the headline to the opening of the body. If the headline leads with a person’s name, the opening should name that person too. A headline about a specific minister over a body that only says “a government official” reads like two different stories.

When that check failed, my pipeline did the obvious thing: it rewrote the headline. That fixed some failures and quietly could not fix others. When I read through the stories that still ended up as drafts, a lot of them had a perfectly good headline. The problem was the body. It had swapped the name for a vague description, “the head of state,” “a candidate,” “a man,” and no amount of headline rewriting was going to put the name back into the body.

The lesson I took from it is general, and I now apply it to any check that compares two things.

A failed comparison does not say which side is wrong

A check like “title matches opening” has two inputs. When it fails, all you know is that they disagree. Picking one side to repair by default is a guess, and if the guess is always the same, you get a repair loop that works on half the cases and wastes a model call on the rest.

So before repairing anything, I ask a second question: which side has the evidence on its side? In this case the evidence is the source material. If the name in the headline appears in the sources, the headline is right and the body left something out. If the name does not appear in the sources, the headline is the problem, and actually it is a worse problem than a mismatch, because it may be invented.

That gives three outcomes instead of one:

  1. Name is in the sources, body omits it. Send the body back for one targeted rewrite with a hint that says exactly what is missing.
  2. Name is not in the sources. Do not touch the body. Asking the body to mention an unsupported name would spread the error into a second place. Leave it as a failure.
  3. Everything matches. Do nothing.

Reuse the rule you already trust

The tempting move is to write a new “is this name really in the sources” function for the repair step. I did not. The publish gate already has a rule for flagging names that are not supported by the sources, and the repair uses that exact function. If the repair and the gate disagreed about what counts as supported, the repair could “fix” a story that the gate would then reject for a different reason, or skip one the gate would have accepted.

The repair decision ended up as a small pure function:

function openingRepairs(titleProblems, titles, bodies, sources) {
	const repairs = [];
	for (const level of Object.keys(titles)) {
		const flagged = titleProblems.some(
			(p) => p.where === `title ${level}` && p.type === 'title does not match opening',
		);
		if (!flagged) continue;

		// Recheck this one level now; the list we were handed may be stale.
		const fresh = titleBodyProblems({ [level]: titles[level] }, { [level]: bodies[level] });
		if (!fresh.some((p) => p.type === 'title does not match opening')) continue;

		const name = leadingName(titles[level]);
		if (!name.every((word) => nameSupported(word, sources))) continue;

		repairs.push({ level, name });
	}
	return repairs;
}

Two details are deliberate. It rechecks the problem for that one level instead of trusting the list it was given, because by the time this runs, earlier steps may have changed the body. And it uses the same nameSupported and titleBodyProblems functions the gate uses, with no new extraction logic. Being pure also makes it easy to test with real failing examples pasted in as fixtures.

One hinted rewrite, then the gate decides

The repair itself is boring on purpose. For each level returned above, it asks for exactly one rewrite of the body with a specific hint, something like “the headline is about this person; name them in the opening sentence.” Then it runs the same cleanup and review steps every body normally goes through, because a repaired body is still a new body and needs the same checks.

for (const { level, name } of openingRepairs(titleProblems, titles, bodies, sources)) {
	const hint = `The headline is about ${name.join(' ')}. Name them in the first sentence.`;
	const revised = await rewrite(level, hint, raw[level]);
	const { text, failures } = await finishLevel(level, revised);

	bodies[level] = toHtml(text);
	qualityFailures = qualityFailures.filter((f) => f.where !== level).concat(failures);
	titleProblems = replaceLevel(titleProblems, level, titleBodyProblems(titles, bodies));

	log(`${level}: repair ${stillMismatched(titleProblems, level) ? 'did not fix' : 'OK'}`);
}

There is no retry loop. If one targeted rewrite does not fix it, a second one with the same hint is unlikely to, and I would rather keep the story as a draft than spend unbounded calls on it. I also make sure the repair replaces only that level’s entries in the failure lists. A repaired body brings its own review results, and the old ones for that level are gone. Results for other levels stay exactly as they were.

Finally, the repair never gets the last word. The final publish gate recomputes every check from the final bodies. If the repair made something else worse, the gate catches it the same way it would catch anything else.

Tradeoffs

  • More moving parts in the pipeline. There is now a step between the title checks and the final gate that can change a body. I kept it contained: one function decides, one loop repairs, and nothing else about prompts or limits changed.
  • Order matters. Anything generated from the body afterwards, like quizzes or summaries, has to be built from the repaired version. I checked that by reading the main flow, and a test covers it.
  • It only fixes one kind of failure. That is intentional. A narrow repair that I can measure is better than a broad “fix whatever is wrong” step that I cannot.

Measure it before you trust it

I wrote down the baseline before merging: about a quarter of story runs were ending as drafts because of this one mismatch. Each repair logs a line I can count, so in two weeks I can check whether that share actually dropped, and whether stories with unsupported names went up, which would mean the repair was hiding a real problem instead of fixing one.

If you have a check that compares two things, look at which side your repair touches. If it is always the same side, try asking the sources which one is wrong first. You might find, like I did, that a lot of your “bad headlines” were fine all along.