I measured my own false-positive rate on 1,728 documents nobody curated

Two populations I didn't choose

The first is the Library of Congress “1000 .gov PDF dataset”, a CC0 random sample of PDFs on .gov domains drawn from their web archive. It was not selected by issuer, not by document type, and not by me. Its published metadata.csv let me check the thing I actually cared about before downloading a single byte: 244 distinct /Producer strings across 1000 files, 144 of them appearing exactly once, spanning PDF 1.1 through 1.7 — Acrobat PDFWriter 3.02, Distiller 4.0, TCPDF, iText, libtiff, Word 2007. Six hand-picked sources cannot produce that spread.

The second I had to build. There is no Brazilian equivalent of the LoC set, so I sampled the Common Crawl index instead — evenly spaced blocks under the br, SURT prefix, which because SURT reverses the hostname covers com.br, gov.br and edu.br together rather than a slice I'd have had to choose. That gave 900 URLs across 421 distinct Brazilian hosts, 335 of them com.br.

990 and 744 files landed. 985 and 743 parsed, with zero failures and zero warning codes. That is worth saying before any rate below: none of what follows is an artefact of files the parser couldn't read.

And the limitation, stated here rather than in a footnote: every document is presumed genuine because it was published by its own issuer, so every signal above info is counted as a false positive. A published .gov PDF that really was altered before publication is scored against me here. That presumption is the weakest joint in the whole study.

The spread was never about the sources

My hypothesis was that the 3.9%-to-36.4% spread across targets was really a producer effect — that a bank statement corpus isn't special because it holds bank statements, it's special because those banks all print through the same two or three toolchains. Inside one population, one country, one document type:

Zero to a hundred percent across fourteen families. The nine-fold spread I'd seen across targets was a producer spread wearing a target's clothes. TCPDF is the clean case: it writes an /OpenAction into every file it produces, so a signal fired on 100% of its output and carried exactly no information.

Two signals were arguing with the parser next to them

The engine reports how many revisions a PDF contains. It is a structural fact, read off the file: one revision means the document was written once and never appended to. Two of my signals were telling the reader a document had been changed after it was created, on files whose revision count said they never were.

That third row is the control, and it's the reason this is a bug rather than a threshold I set badly. incremental-updates is derived from the revision count, so it never disagrees with it. The other two were inferring a second write from an artefact that ordinary producers emit on the first one.

id-inconsistency reads the trailer's /ID array. The spec says the first element is fixed at creation and the second is rewritten on every save, so the two differing is the spec's own record of a re-save. That is what conforming writers do. What writers actually do, on 44.7% of published US government PDFs, is generate both elements fresh the first time.

font-anomalies was worse, because it fired at high — the band my own scoring contract reserves for a signal that on its own establishes a change. Its premise: a generator collects every glyph it needs for a typeface into one subset, so two subsets of one typeface means glyphs were embedded on two separate occasions. Here is a document from the corpus, with the finding it produced:

producer:   Acrobat Distiller 8.1.0 (Windows)
revisions:  1
severity:   high
finding:    Multiple font subsets of the same typeface are embedded
            Calibri (FLJEPG+, FLJFAG+); Cambria-Bold (FLJEIC+, FLJEJC+);
            Cambria-Italic (FLJECP+, FLJEDA+)
            "…the signature left behind when text is added to or replaced
             in an existing PDF with a different tool"

Distiller emits several subsets of one typeface in a single pass as a matter of course. So does every merge — and merging is what iLovePDF, Word's “combine” and a scanner's append button all do to entirely honest documents.

One signal wasn't wrong about its threshold. It was wrong about its premise.

producer-fingerprint was the largest signal in the Brazilian population — 54.5% of 743 documents, 82 of them at high. Its premise, written into the source: statements, invoices and certificates are emitted by server-side libraries, so a consumer tool in the chain is a signal.

Here is what it actually named:

Twenty-six Microsoft Word documents were scored high. Canva's output came back not-low 100% of the time.

The premise holds only if you already know the document claims to be machine-generated. My engine does not know what it is looking at. It receives a file. In an open population the premise inverts completely: a consumer tool in the chain is what an ordinary document has, and a signal that fires on Word is a census, not a finding.

The distinction that survives contact with a real population isn't consumer versus server. It's origin — did this tool create the document, or did it consume one that already existed? Word, Canva, Google Docs, Illustrator, a print driver, an OCR pass over a scan: the file's existence begins with these, so naming them says nothing about whether anything changed afterwards. iLovePDF, Sejda, PDF-XChange, Acrobat's interactive editor: these take an existing PDF as input and write a new one. That is a claim about the file's history rather than its authorship, and it is worth something — though still not proof, because merging, compressing and filling in a form are things honest people do to honest PDFs every day.

There was a second mistake underneath. The signal escalated to high when the document also carried full-page raster imagery, on the theory that this corroborated an edit. In the Brazilian sample that one rule was responsible for 71 of the 82 high firings — because a full-page raster is what a Canva design, an Illustrator export and a scanned page all look like. It marks a document that is a picture. It says nothing about whether the picture was edited.

And one was reporting a table of contents as active content

active-content weighed /OpenAction the same as embedded JavaScript. Across the US sample it fired 213 times: 189 of those were an /OpenAction and nothing else, against 9 for actual JavaScript. One key accounted for 19.2% of that entire corpus, while genuinely executable content sat at 1.2% — the real signal buried under eight times its own volume in noise.

What makes this one cleanly fixable is that the argument is structural rather than statistical. /OpenAction almost always holds a destination — open at page 1, fit to window — which is what most producing tools write by default. When it holds an action instead, that dictionary carries /S /JavaScript or /S /Launch, and the parser's presence scan already walks inline sub-dictionaries and sets those flags from it. So the dangerous case cannot present as openAction alone, which means the alone case is provably navigation.

What changed, and what it was worth

Nothing was suppressed. All four families still emit their findings, with their evidence intact — the conflicting subset prefixes are still listed, the divergent /ID pair is still shown, the /OpenAction is still reported. They emit at info, which scores zero and stays out of the signal count. A reviewer looking at a merged document still sees the fonts. The document just doesn't get a band forced onto it by an inference its own structure contradicts.

Those numbers are a projection, not a re-measurement, and the distinction matters enough to spell out. The original documents were deleted after the findings were written — that is a standing rule of this corpus, not a convenience — so the after-column comes from replaying the scoring function over the stored reports with the new severities. The replay reproduces the engine's current band on 1,728 of 1,728 documents, which is what makes it trustworthy, and it is still not the same thing as running the files again. Both URL manifests survive, so anyone can.

What I did not fix, and cannot yet claim

redaction-exposure fires on 5.1% of the US population and 17.0% of the Brazilian one — text still readable under a drawn box, text sitting under an image placed over it. I cannot classify these. Failed redaction in published government documents is a real and repeatedly documented phenomenon, so an unknown share of them are genuine catches; an image placed over text is also just how a letterhead sits on a page. Resolving it needs a human looking at the documents, which the deletion rule means I no longer have. It is the one number in this study that must not be quoted as a false-positive rate in either direction.

And the larger gap: everything above is specificity. Every document was presumed genuine, so this measures how often the engine is wrong about innocent files and says nothing whatsoever about how often it is right about guilty ones. There is no recall number here because there is no public corpus of documents labelled as tampered — I looked. Producing one means synthesising it: taking genuine documents and applying known edits with a known ground truth. Until that exists, “high precision, unknown recall” is the honest position, and the precision half only became a measured claim this week.

Why publish this

A 68% false-positive rate is not a good number to put on your own product's blog. But the alternative was to keep quoting a rate computed over documents I chose myself, and the entire value of a forensic tool is that its findings mean something when they are inconvenient.

The generalisable lesson isn't about PDFs. It is that a detector measured against a corpus its author assembled is measuring the author. The cost of finding out was two public datasets, an afternoon of fetching, and a rate four times worse than I believed. The cost of not finding out would have been a customer discovering it first, on their own documents, while paying for the privilege.