Make your document generator refuse instead of guessing

  • python
  • automation
  • content

One of my side projects publishes short news stories rewritten for language learners at a few reading levels. Teachers kept asking for something they could print, so I wrote a script that takes one story and produces a worksheet PDF: two versions of the text, a vocabulary table, comprehension questions for each level, and an answer key on the last page.

The PDF part was the easy part. A library like ReportLab will lay out paragraphs and tables all day. The part worth writing about is everything the script does before it draws a single line, because a generator that happily produces a document from bad input is worse than no generator at all. Somebody prints it, hands it to a room full of students, and only then notices the problem.

So the rule I built around: the generator refuses instead of guessing.

Build from content you already trust

The first decision was where the questions come from. I could have generated new ones for each worksheet, but the site already shows a comprehension quiz under every story, and those quizzes have been seen by real readers. Reusing them means the worksheet and the website agree, and I’m not introducing a second, unreviewed source of truth.

The same goes for vocabulary. I already had a hand curated word list for the daily email and a frequency report with glosses. The script picks words that actually appear in the story, curated list first, then the report. It never invents a translation.

That choice shapes the failure modes. If the story doesn’t have enough quiz questions, there’s nothing to fall back on, and that’s correct. The script stops.

Gate the topics you never want to print

News includes stories you would not put in front of a classroom without thinking about it: crime, war, deaths, elections. Most of the time the category on the story is enough to catch these, but categories are assigned by people and pipelines, and they miss things. So the gate has two layers: a blocklist of categories, and a list of patterns checked against the title and the text itself.

The text check has one detail that matters a lot in any language with accents: normalize before you match. A writer might type a word with or without its accent, and a pattern that only matches one spelling quietly lets the other through. Folding to plain lowercase ASCII first means one pattern covers both.

import re, unicodedata

BLOCKED_CATEGORIES = {'politics', 'crime', 'war'}
SENSITIVE = [r'\bguerre', r'\bmort(s|e|es)?\b', r'\battentat', r'\belection']

def fold(s: str) -> str:
    decomposed = unicodedata.normalize('NFD', s.lower())
    return ''.join(c for c in decomposed if unicodedata.category(c) != 'Mn')

def sensitivity(story) -> list[str]:
    hits = sorted(set(story['category'].split('|')) & BLOCKED_CATEGORIES)
    text = fold(' '.join([story['title'], story['text_easy'], story['text_hard']]))
    hits += [p for p in SENSITIVE if re.search(p, text)]
    return hits

The function returns the reasons, not a boolean. When the script refuses, it prints exactly which category or pattern tripped it, so I can tell in two seconds whether it’s a real problem or a false positive like a word that happens to share a stem.

Make the override explicit and boring

A keyword gate will have false positives, and I don’t want to edit the pattern list every time one shows up. The answer is a --force flag, and the important part is what it doesn’t do: it doesn’t silence the check. The check still runs, the reasons are still computed, and the default is still no.

hits = sensitivity(story)
if hits and not args.force:
    sys.exit(f'refusing story {args.story_id}: sensitive ({", ".join(hits)}). '
             'Pick another or pass --force.')

That one line does three things. It exits nonzero, so a scheduled job that wraps the script fails visibly instead of uploading a bad file. It names the story. And it tells the person running it what their two options are. An override you have to type on purpose, after reading why, is a decision. An override buried in a config file is a habit.

Check the shape, not just the presence

The other class of bad input is “technically there, but wrong.” A worksheet with three vocabulary words looks thin. One with thirty doesn’t fit on the page and nobody will learn them. So the script checks counts against the format the teachers expect and stops if they’re off:

if not 8 <= len(vocab) <= 10:
    sys.exit(f'vocabulary list has {len(vocab)} items; need 8 to 10 (use --vocab)')

def questions(story, level, n=5):
    qs = story['quizzes'].get(level, [])[:n]
    if len(qs) < n:
        sys.exit(f'{level}: only {len(qs)} quiz questions; need {n}')
    return qs

Again, the message says what to do next. If the automatic vocabulary pick comes up short, a --vocab file with a hand picked list fixes it, and that file goes into the repo next to the script so the next run of the same story is reproducible.

I’d rather have a worksheet fail at 7 words than ship it with a padding word that doesn’t appear in the text. Silent padding is exactly the kind of guessing this whole design is meant to avoid.

Keep the rest of the run safe to repeat

A few smaller choices round it out:

  • Credentials come from the environment and are never printed. The script can read unpublished stories through an authenticated API, but it works without credentials for public ones, and nothing about the auth ever lands in a log or the PDF.
  • The manifest is an upsert, not an append. Every worksheet gets an entry in a small JSON file that powers the listing page. Regenerating a story removes its old entry before adding the new one, so running the script twice doesn’t list the same worksheet twice.
  • The document says how it was made. The texts are AI assisted rewrites of real reports, so the first page says that and asks teachers to verify anything they plan to teach as fact. A generator that produces something people will hand to others should be honest about its inputs.
  • Watch what you commit. My first pull request for this included a compiled Python cache file. Harmless, but it’s a reminder to add __pycache__/ to .gitignore before the script ever runs inside the repo.

Tradeoffs

  • Keyword gates are blunt. They will flag some harmless stories and miss some harmful ones. That’s acceptable here because a person still picks which story becomes the weekly worksheet. The gate is a seatbelt, not the driver.
  • Refusing costs time. Sometimes the script stops and I have to hand pick vocabulary or choose a different story. That friction is the point, but it means this isn’t a fully unattended pipeline, and I’m fine with that for something a teacher prints.
  • Reusing existing quizzes ties the worksheet to their quality. If a quiz question is confusing on the site, it’s confusing on paper too. The upside is that fixing it once fixes both.

The checklist I’ll reuse

  • Generate from content that’s already reviewed instead of inventing new content per document.
  • Gate sensitive inputs with categories plus normalized text patterns, and return the reasons.
  • Make the override a flag you type on purpose, and keep the check running underneath it.
  • Validate counts and shapes against what the reader expects, and fail with the fix in the message.
  • Exit nonzero on refusal so wrappers and schedulers notice.
  • Upsert any index or manifest so reruns are safe.
  • Say on the document how it was made.