Reference: PDF metadata forensics — what the fields mean
A fraud signal is only worth what its false-positive rate is worth, and the only way to measure that is to run it over a pile of documents you already know are real. So I collected 360 US documents that are authentic by construction — filed in federal court, published by a bank on its own domain, or issued as a government form — and ran every one of them through the engine.
None of them is fraudulent. That is the entire design of the experiment: whatever fires is a false positive, by definition, and the interesting question is not how many fired but why.
The id-inconsistency signal fired on 240 of 255 court-filed
agreements — 94%. Not because the filers modified anything, but because
the court's own electronic filing system re-processes every document that passes
through it. On a large share of the corpus the producer string reads, literally,
Microsoft® Word for Microsoft 365; modified using iText®. A
legitimate federal court document is structurally a modified document.
Mean risk score across those 255 authentic public records: 55. 194 landed in the elevated band and 59 in the high band. Bank-published sample PDFs did worse — mean 81, with 36 of 50 in the high band — because design pipelines leave the same traces, more of them.
The lesson I take from this is narrow and I think it is the important one: a fraud signal without an issuer-baseline notion of "what this pipeline normally does" produces confident nonsense at scale. Not occasionally. On the overwhelming majority of a perfectly ordinary document population.
What the corpus is, and the one that got away
- 255 court-filed agreement PDFs, pulled through CourtListener/RECAP: settlement agreements, employment agreements, leases, loan agreements, asset-purchase agreements. All public record, all filed as exhibits to a docket.
- 50 sample and marketing PDFs published by 39 financial institutions on their own domains.
- 55 government forms and templates — 36 PDF, 19 OOXML.
- 360 US files in total, part of a 705-document run that day across six targets. Zero parse failures across all 705.
EDGAR was the first idea, and it was wrong. SEC filers attach material contracts as exhibits, which sounds like an enormous free corpus of real-world agreements. It is — but since the EDGAR modernisation those exhibits are HTML. You can pull thousands of them and get almost no PDFs at all. Court dockets are the opposite: the filing system produces PDF by construction, because that is what a court clerk stamps. I abandoned EDGAR after an afternoon and I would abandon it again.
No third-party document appears anywhere in this post, and none was kept. The engine ran locally, in-process, so no fetched file ever transited a network to a hosted API. The raw documents were deleted the same day the findings were written; what remains is a manifest of source URL, SHA-256 and timestamps per file, which is enough to reproduce every number here and not enough to reconstruct anybody's document. Everything below is an aggregate.
Finding 1 — the filing system is the modifier
A PDF's trailer carries an /ID array of two byte strings. The first
is set when the file is created and is meant never to change again; the second is
rewritten on each save. When a file has been written once, they are identical.
When they differ, something wrote the file after it was created. That is the whole
of the id-inconsistency signal, and it is described field by field in
the metadata forensics guide.
It fired on 240 of 255 filings. The cause is not subtle once you look at the producer strings: CM/ECF, the federal courts' electronic filing system, re-processes every document submitted to it — and the tool it re-processes them with signs its work.
The five most common producer strings in the corpus are variations on the same
sentence: a word processor, a semicolon, and modified using iText.
One large group carries iText alone. Another sizeable group carries an Acrobat
paper-capture plug-in — scanned exhibits, OCRed on the way in.
This is not a court-specific curiosity. It is the general shape of the problem: a document that has been through any institutional intake pipeline — e-filing, document management, accessibility remediation, redaction review — comes out the other side structurally indistinguishable from a document somebody edited. The bytes cannot tell you which it was. Only knowing the pipeline can.
Finding 2 — the noise floor is a score of 55
Signals aggregate into a score, and the score into a band. Here is what a corpus of certainly-authentic federal court exhibits scores.
The other signals ride along with the first, and they are plausibly the same story told three more ways.
| signal | fired on | the benign explanation |
|---|---|---|
id-inconsistency | 240 / 255 | the filing system re-writes the file |
hybrid-page | 100 / 255 | scanned exhibit with an OCR text layer |
producer-fingerprint | 93 / 255 | a stamping library in the producer string |
font-anomalies | 36 / 255 | the stamp embeds its own font beside the author's |
Every one of those findings is true about the bytes. The file really was written twice. There really is text drawn over a page image. A second font subset really was added by a second tool. The parser is not wrong. The score is wrong, because it is answering a question about intent using evidence that only describes mechanism.
Finding 3 — marketing PDFs are noisier than contracts
The 50 bank-published sample PDFs, from 39 issuers, scored a mean of 81, with 36 of 50 in the high band. That makes bank marketing collateral the noisiest legitimate document class I have measured — noisier than court filings, by a wide margin.
The producer strings say why: Adobe PDF Library across three major versions
accounts for the largest single group, alongside InDesign, Canva and the usual
office suites. A brochure gets written in one tool, laid out in a second,
exported, re-exported for accessibility, and re-exported again for the web.
id-inconsistency fired on 42 of 50, metadata-mismatch on
28, producer-fingerprint on 28.
This lands on top of a result from a different corpus, published alongside this one: bank website PDFs have no usable issuer fingerprint — the producer strings are all over the place. Put the two together and the conclusion is uncomfortable in a specific way. The document class that is easiest to collect from a bank is simultaneously the one with the least issuer signal and the most editing noise. It is the worst possible thing to build a baseline out of, and it is what is lying around.
Finding 4 — a government form contains JavaScript on purpose
The 55 government forms and templates scored a mean of 48, which is the tamest
number in the run — but they surface a benign class worth naming outright.
active-content fired on 18 of 55.
The cause is entirely mundane: fillable forms are built in LiveCycle Designer as XFA, or carry AcroForm JavaScript for field validation, date pickers and arithmetic. A tax form that computes a subtotal contains JavaScript because somebody wanted the subtotal computed.
"This PDF contains active content" is a true statement and a useful one when the document is supposed to be a static statement. On a government fillable form it is a description of the form. The malware-signals guide makes the same distinction from the other direction: the presence of an embedded action is not the finding, what the action does is.
What this changes in the engine
The obvious fix is not "raise the threshold". Raising a threshold to swallow a known-benign pattern also swallows the real thing when it happens to score similarly, and you cannot tell afterwards which one you suppressed.
What the data argues for is modifier-aware severity: a small set
of recognised pipelines, each with an expectation attached. If a producer string
names a court e-filing stamper, then a changed /ID pair is the
expected outcome and should carry the weight of an expected outcome — while a
second, later modification on top of the stamp is exactly as interesting
as it ever was, and stays interesting.
That is the same machinery as an issuer baseline, applied to the processor rather than the issuer, and it is the strongest false-positive lever this corpus has produced. It is also, conveniently, cheap: the pipelines that matter are few, they announce themselves in the producer string, and the list is short enough to audit by hand.
Until that ships, the engine's honest behaviour is what it already does — report the signal and its severity, and leave the band alone. A signal you can read is recoverable. A verdict you cannot see inside is not.
The limits you should hold this against
This corpus is not a random sample of anything. It is what CourtListener happened to hold for a handful of agreement-shaped full-text queries, what 39 institutions happened to publish, and what a set of government sites happened to offer. Percentages here describe these 360 files. They do not describe US public documents, and I am not going to pretend otherwise.
These are false positives of the score, not of the parser. Every finding above is factually correct about the file. That distinction matters because the fix is not "detect less" — it is "explain more". A signal that fires with a named benign cause attached is more useful than a signal that stays quiet.
The reverse error is unmeasured and stays unmeasured. A document fabricated cleanly from scratch, or laundered through a print-to-PDF pass, comes out of this engine looking better than most of the court record does. Nothing here says anything about recall, and I have no defensible number to offer for it.
This reports signals, never a verdict. "The trailer's
/ID pair does not match" is a statement about bytes. "This contract
was tampered with" is a statement about a person, and 240 authentic court filings
are the reason a parser does not get to make it.
Run the same parser on a file of your own
Every number above came out of the engine behind the
free checker, running locally over files that were deleted the
same day. Drop in a PDF and you get the same report this calibration was built
from — the trailer /ID pair, the producer and creator strings, the
revision chain, the fonts, the page composition. No account, nothing stored.
If you are integrating this and the noise floor above is your problem too, comparing a document against one you already trust is the version of this measurement that does not need a baseline anybody else built.