Put every verification rule into the generation prompt too

  • ai
  • llm
  • reliability

I already check generated text in code and retry exactly once when something fails. That pattern has saved me from a lot of bad drafts. It does not, by itself, stop the same failures from showing up morning after morning.

The checker kept catching the same handful of problems: a year the sources never wrote, a title that hardened a soft claim, a work title chopped in half, a length floor met by inventing filler. The one rewrite almost always fixed that draft. The next run failed the same way.

The prompt and the gate had drifted apart. The gate knew the rules. The prompt mostly hoped.

The prompt is a preview of the checker

A verification function is the source of truth. That does not mean the prompt gets to stay vague. Every failure mode the checker can name should also appear, in plain language, in the generation instructions. Otherwise you burn the single retry on mistakes you already know how to describe.

I rewrote the system prompt around a short mission and a numbered list of absolute rules, then put a checklist in the user prompt right next to the immutable facts. The checklist is not inspiration. It is the same list the gate will walk after the model replies.

Roughly:

function yearLockInstruction(years) {
	const ys = uniqueValidYears(years);
	if (!ys.length) {
		return 'No year appears in the sources. Write none. Do not invent one.';
	}
	if (ys.length === 1) {
		return `The source year is ${ys[0]}. It is immutable. Never substitute another.`;
	}
	return `The only source years are ${ys.join(', ')}. Change none. Add none.`;
}

function rewriteUserPrompt({ sources, minWords, sourceYears }) {
	return `
SOURCES:
${sources}

IMMUTABLE YEAR:
${yearLockInstruction(sourceYears)}

CHECKLIST BEFORE YOU WRITE:
1. One event only, from source 1. No names from general knowledge.
2. Keep claim strength exact. Do not harden or soften hedging words.
3. If today's event has no year in the sources, write no year.
4. Copy work titles in full, quoted exactly as the sources write them.
5. Reach at least ${minWords} words by explaining source facts more simply, never by adding facts.

Write the article now.
`.trim();
}

Two details mattered more than the formatting.

First, absence is allowed. The title checker used to treat a missing year as a defect. That pushed the model to invent “2026” for an event the wires only dated as “Thursday.” The fix was one sentence on the checker side: a missing year is never a failure. An invented year still is. The prompt and the gate now agree that silence is correct when the sources are silent.

Second, claim strength is a first class rule, not a vibe. Soft language in the source (“could,” “according to,” “says”) has to survive into the draft and the title. Hardening “no immediate impact on jobs” into “jobs at risk” is a factual change, even when every proper noun is still correct. Softening a strong verb the other way is the same bug. Naming that in the prompt cut a whole class of title failures.

Keep the checker strict anyway

Aligning the prompt does not replace the gate. Models still ignore instructions. The year lock still runs in code after generation. Titles still get scanned for proper nouns and dates that appear in neither the verified body nor the sources. A draft that fails still stays a draft.

What changes is the success rate of the first pass. The retry becomes the exception instead of the routine. That matters when each generation costs money and time, and when a failed draft blocks a slot until the next run.

I also stopped asking the model to “fill” a length floor without telling it how. “Write at least N words” with no method is an invitation to invent. The prompt now says: reach the floor by explaining the source facts more simply. The checker still measures the count. The prompt just stops pointing at the wrong way to get there.

A small asymmetry worth keeping

The prompt can be longer than the checker. The checker has to stay cheap, pure, and unit tested. The prompt can afford concrete counterexamples (“do not turn an adjective into a country name”) that would be awkward to encode as regex. Put the expensive nuance in the instructions. Put the cheap, decisive measurements in code.

When a new failure mode shows up twice, I do both:

  1. Add a check (or tighten an existing one) so the bad draft cannot publish.
  2. Add the same rule to the prompt (and the checklist) so the next first draft is less likely to hit it.

Doing only step 1 leaves you retrying forever. Doing only step 2 leaves you hoping. Together they make the pipeline boring, which is the goal.

When this is worth it

If you generate once and eyeball the result, a careful prompt is enough. If you generate on a schedule into a gate that can refuse to publish, keep the prompt and the gate in lockstep. Every rule the gate enforces should be visible to the model before it writes a word. The checker remains the authority. The prompt becomes a preview of that authority, written in the language the model actually reads.