What this engine gets wrong, and how we know

How to read a number on this page

Everywhere else on this site, a guide page is forbidden from carrying a number that will go stale — dated claims belong on the blog, where the date bounds them. This page is the deliberate exception, and it is contained in two ways rather than left to good intentions.

• Every figure carries the engine version it was measured on and the date. A rate measured on 1.17.0 is a claim about 1.17.0. Where a later release moved it, both numbers are here and so is the release that moved them.
• A test in CI holds this page to the shipped engine. It asserts that the version this page calls current is the version in the engine's own source, so an engine release cannot ship without someone having come back and re-read what is written here. The engine running in production today is 1.28.0.

Two words are used precisely throughout. A measurement means the engine was run over the files. A projection means the scorer was replayed over reports that were already stored, which is weaker, and it is labelled as such every time it appears.

False positives, on 1,728 documents nobody curated

Every corpus this engine had before August 2026 was one we chose — court filings, public procurement contracts, bank statement samples — and a rate computed over documents you picked is a fact about the picking. So two populations nobody here assembled were measured instead:

• 985 US .gov PDFs from the Library of Congress “1000 .gov PDF dataset” (CC0).
• 743 Brazilian web PDFs drawn from evenly spaced Common Crawl index blocks, across 421 Brazilian hosts.

1,728 documents, 0 parse failures. Every one was published by the organisation that made it, so each is presumed genuine and every score above low counts against us. The measurement is dated 4 August 2026 and it is what forced the engine 1.17.0 re-specification: four signal families were gated or re-scoped because they were asserting “changed after creation” about files whose own revision count says they were written exactly once.

Left of each arrow: measured, on the engine as it stood before 1.17.0, 4 August 2026. Right of each arrow: engine 1.17.0, 4 August 2026 — a projection, not a re-measurement.

This has not been re-measured at the whole-population level since 1.17.0, and the honest reading of the gap is that the number above is, if anything, high for the engine you would call today: 1.19.0, 1.22.0 and 1.23.0 each removed firings on these same populations rather than adding them, and 1.27.0's new check ships at low, which cannot on its own lift a document out of the low band. The write-up of the original run, with the four families and what each was doing wrong, is here.

Recall, per forgery operation

Specificity is the easy half: genuine documents are published by the thousand. The other half is harder, because no public corpus labels documents as tampered — the academic forgery sets are pixel-level image forensics, and the PDF corpora hold malformed and malicious files, which is a different problem. A malicious PDF attacks the reader; a forged one lies to it.

So the positives were built. 160 genuine documents, published by their own issuers and drawn from the two populations above, every one scoring low with a single revision beforehand — so there is no false-positive confound inside the result — with eight known edits applied to each, giving 1,006 forged files. That they are forgeries and not merely broken files was checked three ways: qpdf --check reported 944 clean, 62 warnings and 0 broken, with every warning tracing to a source that already had them; the pages were rendered and inspected; and each covering box was aimed at a text run the parser had retained rather than at a guessed band, because a box over white space hides nothing.

Measured, engine 1.23.0, 5 August 2026 — 1,006 forged files from 160 genuine sources, re-fetched byte-identical, with 0 confounds: not one of the 160 sources scored above low before it was edited. First run on 1.18.0 (4 August 2026); re-measured and identical to the decimal on 1.19.0, 1.22.0 and 1.23.0.

Stated in one sentence: a naive forgery is caught every time; a forgery composited through a Form XObject is caught roughly nine times in ten; laundering the whole file through a rewrite degrades detection to about two thirds. The Form XObject row was 0.7% before engine 1.18.0 (4 August 2026) — the walker stopped at the form boundary, so a white box drawn inside one was never compared with the text underneath it. That was disclosed on every report at the time, which was honest and did not make it any cheaper.

It also bounds what the number means: these are the forgeries we thought of. Eight operations, applied by a script, are not an adversary.

The null result: a cleanly regenerated file

Every recall figure above starts from a genuine document and applies an edit — the class byte-level forensics exists for. The class nobody had measured is the one that was never edited at all: a document laid out and printed fresh, with fraudulent content in it from the first byte.

Twenty of them were built — 8 statement and payslip lookalikes laid out in HTML and printed by headless Chromium, 6 already-caught forgeries flattened to images with pdftoppm, and 6 written byte by byte in one clean pass with a server-library producer string or none at all.

This is not a defect, and we are not going to fix it with a threshold. Nothing was edited, so there is no edit to find. The engine behaved exactly as designed under the same test: Chromium's font subsets and fresh /ID pairs were emitted at info, scoring zero, which is precisely what the 1.17.0 recalibration was for — a pre-1.17 engine would have “caught” all eight Chromium prints and been wrong about why.

What it means for you is a scoping rule rather than a caveat: this engine answers was this file changed after it was made. It does not answer is this document true. A clean regeneration is the boundary between those two questions, and the answer to the second one has to come from somewhere else — an issuer baseline, a bank feed, a phone call. Whether private issuer baselines catch this class is not yet measured, and it is the next benchmark.

The one family that survives laundering

redaction-exposure asks whether text somebody meant to remove is still readable underneath the box drawn over it. It matters disproportionately for one reason: it is the only signal family that survives a whole-file rewrite — a rewrite destroys the revision history every other family reads, but text under a box is still text under a box. It accounted for 38.3% of all detections across the forgery set, and roughly 41% of what is left after a rewrite.

It was also, for a while, our worst false-positive source, and fixing it was worth this much work precisely because of the paragraph above. The method did not involve reading a single document for meaning: render the page, and look at the pixels where the supposedly-hidden run sits. Text under a box is a flat block; text that is not hidden has the contrast of glyphs. Across 1,693 covered runs nothing sat between 0.7 and 12.2, so the threshold is a gap in the data rather than a tuned constant.

Recall is unchanged at every one of those steps — the per-operation table above was re-measured after each and is identical to the decimal, with 0 forgeries lost. And the finding that matters most for anyone reading a report: across those 1,728 published documents this family found no failed redaction, in either population. Roughly nine in ten of its original firings were demonstrably wrong or vacuous.

What remains is stated rather than hidden. The residue after 1.19.0 was 18 US documents: twelve genuine false positives from non-rectangular clip paths this parser does not model, and six that really do hide text — every one a layout artifact, a chart label behind a plot fill, a map over its own draft paragraph, a template footer. The image-cover demotion in 1.23.0 was decided the same way: across 1,006 forgeries, image covers accounted for zero detections against 385 from filled boxes, so the observation stays in the report at info and stops carrying a score. The redaction checker runs exactly this family on a file you hold.

The same number, per toolchain

A population-wide false-positive rate is the right number for deciding whether to buy this. It is the wrong number for reading one report, because the rate is not evenly spread — it is largely a property of the software that wrote the file. Inside a single population, a signal's firing rate ran the whole way from 0% to 100% depending on which software wrote the document. That is what an earlier finding — one corpus far noisier than another — turned out to be: a producer effect wearing a target's clothes.

So the per-toolchain version of this page's headline number already exists: the producer-string reference lists the 40 /Producer strings common enough in that corpus to have been measured, and its last column is our own false-positive rate on that toolchain's genuine output, never a fraud rate. Every document behind it was published by the organisation that made it, so each score above low counts against us there too.

Same corpus, scored on engine 1.22.0: 432 distinct producer strings across the 1,728 documents, 252 of them appearing exactly once, which is the most useful fact on that page — the long tail is the normal case, and a string you have never seen before is not evidence of anything.

What is still not measured

• Whether issuer baselines catch a clean regeneration. That is the counter we sell against the null result above, and it has not been benchmarked. It will be measured and published either way.
• A whole-population false-positive rate on the current engine. What exists is the 1.17.0 projection above plus per-family re-measurements at 1.19.0, 1.22.0 and 1.23.0. The full re-run is reproducible from the surviving manifests and has not been done.
• Recall against a real adversary. The positives are built by a script from eight operations we chose. Nothing here says how the engine performs against a forger who reads this page.
• Whether a forger would hide text in a layer. A PDF can carry content in an optional-content group switched off by default: present in the file, absent from the screen. Engine 1.28.0 reads and retains those runs and judges none of them, and it established that glyph-anomalies, br-identifier-checksum and running-balance-break read such text as though it were on the page. Measured across the same 1,727 documents on 2026-08-06: 85 declare optional content (4.92%), 3 carry a group that is off by default (0.17%), and none paints text into one — so on this population no figure above is affected, and the three families never once read a hidden line. What that does not establish is the thing worth knowing: these are documents nobody was hiding anything in. A null result on published files is evidence about accidents, not about adversaries.
• Image metadata is trivially strippable. exiftool -all= removes every identity and timestamp finding the image engine makes. A match is strong evidence; an absence is none, and every image finding says so in its own prose.
• The Office engine does not render the document. Text hidden by being coloured to match its background, sized to zero, or covered by a shape is not detected there — the paint-order pass that makes redaction-exposure work has no OOXML equivalent yet. It is the biggest hole in the newer medium, and it is disclosed on every Office report rather than left to be inferred.
• Encrypted PDFs are disclosed, not attacked. A password-protected file is reported as “not content-analysed” rather than scanned as ciphertext and reported as clean.

The number nobody in this market publishes

The competitive landscape was read directly on 31 July 2026: Resistant AI, Inscribe, Snappt, Ocrolus, Truv, Veryfi, DocuClipper, VerifyPDF and HTPBE. Several publish a headline accuracy figure in the 99% range with no methodology published, no stated denominator and no ground truth — which is why none of those figures is repeated here.

Not one of them publishes how often its system flags a genuine document. That absence is the reason this page exists. It is not a claim that our engine is better than theirs: several of them do things this one cannot, at a scale this one is not built for. It is a claim about what you can check. A recall-flavoured number with no denominator cannot be argued with, only believed; a false-positive rate on a named, public, uncurated population can be reproduced by anyone who fetches the same manifests.

Even the strongest public benchmark in the adjacent space leads with the false accept rate. False-reject behaviour on genuine documents is the thing every operations team actually pays for, and it is the thing nobody quotes.

Sources for every figure on this page are tracked in this product's own repository: the engine release notes in CHANGELOG.md (1.17.0, 1.18.0, 1.19.0, 1.22.0, 1.23.0, 1.27.0), the “Honest gaps” section of STATUS.md, and the evidence compilation in docs/EVIDENCE.md §5.2 for the competitive reading above. Per-document corpus findings are not published: the corpus rules delete the documents after a run, and publishing a list of published organisations' filenames beside a risk band is not something a measurement needs.