Build your download bundles so the same inputs give the same bytes

  • python
  • automation
  • reliability

Sooner or later, most content sites want a “download everything” button. On a side project of mine that publishes a printable PDF every week, teachers kept grabbing each week one at a time, so I added a single ZIP that holds all of the current ones. The ZIP is generated by a script and committed to the repo, because the site is a theme deployed from git and there’s no build server to make it on the fly.

Writing a script that zips some files takes five minutes. Writing one that I can rerun every week without making a mess took a little more thought, and the extra thought is the part worth sharing.

The problem with a naive ZIP

The obvious version looks like this:

import zipfile
from pathlib import Path

with zipfile.ZipFile('pack.zip', 'w') as zf:
    for pdf in Path('worksheets').glob('*.pdf'):
        zf.write(pdf, pdf.name)

It works, and the output opens fine. But run it twice with no changes and you’ll usually get two different files. A ZIP stores a modification time and permission bits for every entry, and zf.write copies them from the file system. A fresh clone, a different machine, or a tool that touched a file all change those values. The glob order isn’t guaranteed either.

That matters more than it sounds when the file lives in git:

  • Every rerun produces a binary diff, even when nothing a reader would see has changed.
  • Review gets noisy. You can’t tell from the pull request whether the pack actually changed.
  • Caches and CDNs see a new file and throw away a perfectly good copy.
  • You lose the cheap sanity check of “I rebuilt it and the hash matched.”

The goal is simple to state: the same inputs should give the same bytes. If the PDFs didn’t change, the ZIP shouldn’t either.

Decide what the inputs are

The first step was deciding where the list of files comes from. The site already had a small JSON manifest with one row per week: the story it belongs to, the week date, and the path to the PDF. Other parts of the site read that same file to decide which pages get special treatment, so it was already the source of truth.

So the pack script reads the manifest instead of scanning a folder. That gives me three things for free. The order is the manifest’s order, not whatever the file system returns. A stray PDF sitting in the folder doesn’t sneak into the download. And when a row points to a PDF that isn’t there, the script stops instead of quietly shipping a pack with a hole in it.

def pack_rows(rows, theme_dir):
    files = []
    for row in rows:
        if not row.get('pdf'):
            continue
        path = theme_dir / row['pdf'].lstrip('/')
        if not path.is_file():
            sys.exit(f'missing worksheet PDF: {path}')
        y, m, d = (int(p) for p in row['week_of'].split('-'))
        files.append((path.name, (y, m, d, 0, 0, 0), path.read_bytes()))
    if not files:
        sys.exit('no PDFs in the manifest')
    return files

Notice where the timestamp comes from. It’s not the file’s modification time. It’s the week date from the manifest, at midnight. That date is meaningful to anyone who opens the ZIP (the entries sort and display sensibly), and it only changes when the data changes.

Pin everything the format lets vary

With the inputs settled, the writer sets each entry’s metadata by hand instead of letting zipfile copy it from disk:

def write_pack(files, dest):
    dest.parent.mkdir(parents=True, exist_ok=True)
    tmp = dest.with_suffix(dest.suffix + '.tmp')
    with zipfile.ZipFile(tmp, 'w') as zf:
        for name, stamp, data in files:
            info = zipfile.ZipInfo(filename=name, date_time=stamp)
            info.compress_type = zipfile.ZIP_DEFLATED
            info.external_attr = 0o644 << 16
            zf.writestr(info, data)
    tmp.replace(dest)

Building a ZipInfo yourself is the key move. You choose the name (just the file name, no leaked directory structure from my machine), the timestamp, the compression method, and the Unix permission bits. 0o644 << 16 is how ZIP stores “readable by everyone, writable by the owner” in the upper half of that field. Leave it out and you get whatever the defaults are, which can differ between Python versions and platforms.

The compression level is the one thing I didn’t pin, and that’s a deliberate tradeoff. Deflate output is stable for the same library version, which is all I need for a file rebuilt in one place. If you need byte for byte matches across very different environments, pin the level and the Python version too, or use a dedicated reproducible archive tool.

Write to a temp file, then swap

The last line, tmp.replace(dest), is small but it earns its place. The script writes the whole archive to a sibling temp file and only renames it over the real path once it’s finished. A rename on the same file system is atomic, so anything reading the path sees either the old complete ZIP or the new complete one, never half of one. If the script dies partway through, the published file is untouched and the leftover .tmp is obvious.

That’s the same habit I use for any generated file that something else reads: build it somewhere else, then move it into place.

Keep the URL boring

The pack always lands at the same path, with the same name, no date in it. The page that links to it never has to change, and neither do any links people have saved. The only thing that changes is the contents, and because the build is reproducible, that only happens when a new week is actually added.

The script that produces each weekly PDF calls the pack script right after it updates the manifest. So there’s one command to add a week, and the bundle can’t drift out of sync with the list of weeks.

The test that proves it

The check I care about is cheap: build the pack twice and compare.

python3 scripts/make-pack.py && sha256sum pack.zip
python3 scripts/make-pack.py && sha256sum pack.zip

If the two hashes differ, something nondeterministic crept in, usually a timestamp or an ordering that came from the file system. I also list the archive (unzip -l) in review to confirm it holds exactly the files the manifest says it should.

When to bother

You don’t need any of this for a throwaway export a user downloads once. It pays off when a generated artifact is committed, cached, compared, or rebuilt on a schedule. In those cases, a file that changes only when its contents change is easier to review, easier to trust, and kinder to every cache between you and the reader. It’s about fifteen extra lines, and I’d write them again.