What this engine gets wrong, and how we know

How to read a number on this page

Everywhere else on this site, a guide page is forbidden from carrying a number that will go stale: dated claims belong on the blog, where the date bounds them. This page is the deliberate exception, and it is contained in two ways rather than left to good intentions.

• Every figure carries the engine version it was measured on and the date. A rate measured on 1.17.0 is a claim about 1.17.0. Where a later release moved it, both numbers are here and so is the release that moved them.
• A test in CI holds this page to the shipped engine. It asserts that the version this page calls current is the version in the engine's own source, so an engine release cannot ship without someone having come back and re-read what is written here. The engine running in production today is 1.37.0. That release added the nineteenth signal family, extraction-anomalies: pages whose extracted text cannot be trusted to be the text a reader sees, measured on 5,168 published documents before any severity shipped (docs/CORPUS-PHASE0-EXTRACTION-2026-08-31.md in the repository). The release before it, 1.36.0, changed nothing in the engine: it is a security-review batch, cut as a release only so the fixes reach the published MCP package on npm. That batch locks recovered redaction text that is shaped like a prompt injection out of inspect_document's reply, adds an optional TAMPERLENS_ALLOWED_DIRS read allow-list that confines a local path to named directories, and hardens the compare_documents upload; the billing webhook, which is server-side and outside the engine, now gates fulfilment on Stripe's payment_status. Every report is byte-identical to a 1.35.0 one; the version moved only because the published MCP package is pinned to the engine version. The release before it, 1.35.0, also changed nothing in the engine: it gave the MCP server's single-file tools a url alternative to a local file path, so an agent can point them at an http or https link and the server fetches the document over its own machine's network before sending the bytes on. Before that, 1.34.0 added a measured I/O cost block to every report and a cheap ?mode=triage scope for agents deciding whether a document is worth a full inspection. It was measured on the same 5,168-document corpus first: triage expands 9% of a full parse's decompressed bytes, reproduces the structural, metadata and signature families count-for-count, and is silent on the page-content families, which the report's triage block states plainly, never left to be inferred. It moved no signal severity and no specificity figure below, and each number there stays attributed to the version it was taken on. Before that, 1.33.0 added two detections to the active-content family: a dictionary name concealed by a hex-escape, and the /AA additional-actions key, each measured on the same corpus before it was given a severity. 1.31.0 folded three Office signal families to one signal each, which the report invariant had always said they were; it changed no PDF and no image behaviour either, and every figure below is a PDF-corpus figure. Each number stays attributed to the version it was taken on. 1.30.0 changed how revisions are counted when an appended trailer omits /Prev, the PNG chunk cap, the generator-name patterns and the boleto due-date factor, and whether any document in the two populations carries the /Prev-less shape has still not been checked.

Two words are used precisely throughout. A measurement means the engine was run over the files. A projection means the scorer was replayed over reports that were already stored, which is weaker, and it is labelled as such every time it appears.

False positives, on 1,728 documents nobody curated

Every corpus this engine had before August 2026 was one we chose: court filings, public procurement contracts, bank statement samples, and a rate computed over documents you picked is a fact about the picking. So two populations nobody here assembled were measured instead:

• 985 US .gov PDFs from the Library of Congress “1000 .gov PDF dataset” (CC0).
• 743 Brazilian web PDFs drawn from evenly spaced Common Crawl index blocks, across 421 Brazilian hosts.

1,728 documents, 0 parse failures. Every one was published by the organisation that made it, so each is presumed genuine and every score above low counts against us. The measurement is dated 4 August 2026 and it is what forced the engine 1.17.0 re-specification: four signal families were gated or re-scoped because they were asserting “changed after creation” about files whose own revision count says they were written exactly once.

Left of each arrow: measured, on the engine as it stood before 1.17.0, 4 August 2026. Right of each arrow: engine 1.17.0, 4 August 2026, a projection rather than a re-measurement.

This has not been re-measured at the whole-population level since 1.17.0, and the honest reading of the gap is that the number above is, if anything, high for the engine you would call today: 1.19.0, 1.22.0 and 1.23.0 each removed firings on these same populations rather than adding them, and 1.27.0's new check ships at low, which cannot on its own lift a document out of the low band. The write-up of the original run, with the four families and what each was doing wrong, is here.

Recall, per forgery operation

Specificity is the easy half: genuine documents are published by the thousand. The other half is harder, because no public corpus labels documents as tampered. The academic forgery sets are pixel-level image forensics, and the PDF corpora hold malformed and malicious files, which is a different problem. A malicious PDF attacks the reader; a forged one lies to it.

So the positives were built. 160 genuine documents, published by their own issuers and drawn from the two populations above, every one scoring low with a single revision beforehand (so there is no false-positive confound inside the result), with eight known edits applied to each, giving 1,006 forged files. That they are forgeries and not merely broken files was checked three ways: qpdf --check reported 944 clean, 62 warnings and 0 broken, with every warning tracing to a source that already had them; the pages were rendered and inspected; and each covering box was aimed at a text run the parser had retained rather than at a guessed band, because a box over white space hides nothing.

Measured, engine 1.23.0, 5 August 2026: 1,006 forged files from 160 genuine sources, re-fetched byte-identical, with 0 confounds: not one of the 160 sources scored above low before it was edited. First run on 1.18.0 (4 August 2026); re-measured and identical to the decimal on 1.19.0, 1.22.0 and 1.23.0.

The Form XObject row was 0.7% before engine 1.18.0 (4 August 2026): the walker stopped at the form boundary, so a white box drawn inside one was never compared with the text underneath it. That was disclosed on every report at the time, which was honest and did not make it any cheaper.

It also bounds what the number means: these are the forgeries we thought of. Eight operations, applied by a script, are not an adversary.

The null result: a cleanly regenerated file

Every recall figure above starts from a genuine document and applies an edit. The class byte-level forensics exists for. The class nobody had measured is the one that was never edited at all: a document laid out and printed fresh, with fraudulent content in it from the first byte.

Twenty of them were built: 8 statement and payslip lookalikes laid out in HTML and printed by headless Chromium, 6 already-caught forgeries flattened to images with pdftoppm, and 6 written byte by byte in one clean pass with a server-library producer string or none at all.

This is not a defect, and we are not going to fix it with a threshold. Nothing was edited, so there is no edit to find. The engine behaved exactly as designed under the same test: Chromium's font subsets and fresh /ID pairs were emitted at info, scoring zero, which is precisely what the 1.17.0 recalibration was for. A pre-1.17 engine would have “caught” all eight Chromium prints and been wrong about why.

What it means for you is a scoping rule rather than a caveat: this engine answers was this file changed after it was made. It does not answer is this document true. A clean regeneration is the boundary between those two questions, and the answer to the second one has to come from somewhere else: an issuer baseline, a bank feed, a phone call. Whether private issuer baselines catch this class is not yet measured, and it is the next benchmark.

The one family that survives laundering

redaction-exposure asks whether text somebody meant to remove is still readable underneath the box drawn over it. It matters disproportionately for one reason: it is the only signal family that survives a whole-file rewrite, a rewrite destroys the revision history every other family reads, but text under a box is still text under a box. It accounted for 38.3% of all detections across the forgery set, and roughly 41% of what is left after a rewrite.

It was also, for a while, our worst false-positive source, and fixing it was worth this much work precisely because of the paragraph above. The method did not involve reading a single document for meaning: render the page, and look at the pixels where the supposedly-hidden run sits. Text under a box is a flat block; text that is not hidden has the contrast of glyphs. Across 1,693 covered runs nothing sat between 0.7 and 12.2, so the threshold is a gap in the data rather than a tuned constant.

Recall is unchanged at every one of those steps. The per-operation table above was re-measured after each and is identical to the decimal, with 0 forgeries lost. And the finding that matters most for anyone reading a report: across those 1,728 published documents this family found no failed redaction, in either population. Roughly nine in ten of its original firings were demonstrably wrong or vacuous.

What remains is stated rather than hidden. The residue after 1.19.0 was 18 US documents: twelve genuine false positives from non-rectangular clip paths this parser does not model, and six that really do hide text, every one a layout artifact, a chart label behind a plot fill, a map over its own draft paragraph, a template footer. The image-cover demotion in 1.23.0 was decided the same way: across 1,006 forgeries, image covers accounted for zero detections against 385 from filled boxes, so the observation stays in the report at info and stops carrying a score. The redaction checker runs exactly this family on a file you hold.

The same number, per toolchain

A population-wide false-positive rate is the right number for deciding whether to buy this. It is the wrong number for reading one report, because the rate is not evenly spread. It is largely a property of the software that wrote the file. Inside a single population, a signal's firing rate ran the whole way from 0% to 100% depending on which software wrote the document. That is what an earlier finding, one corpus far noisier than another, turned out to be: a producer effect wearing a target's clothes.

So the per-toolchain version of this page's headline number already exists: the producer-string reference lists the 40 /Producer strings common enough in that corpus to have been measured, and its last column is our own false-positive rate on that toolchain's genuine output, never a fraud rate. Every document behind it was published by the organisation that made it, so each score above low counts against us there too.

Same corpus, scored on engine 1.22.0: 432 distinct producer strings across the 1,728 documents, 252 of them appearing exactly once, which is the most useful fact on that page. The long tail is the normal case, and a string you have never seen before is not evidence of anything.

The measurements that failed

The house rule here is that no check ships before it is measured against the corpus, and when the measurement fails, what gets published is the failure, not the check. Several planned checks do not exist in this engine today because a measurement said no, and one gate could not be read at all. Leaving these off this page would make every flattering number above cheaper.

The first run of that script deserves its own sentence, because it read like an answer: it reported a clean 0 across all 3,822 documents, and the 0 was an artifact, the scan ran over compressed stream bytes it could not see into. A measurement that cannot see the thing it measures returns zero, and zero is indistinguishable from a result.

One published measurement was itself wrong, and the retraction is part of the record. The first phase-0 run for certification permissions enumerated 8 of the corpus's 14 targets, 915 PDFs where the corpus held 3,842, and reached three conclusions a full run showed to be false, including “nobody in this corpus certifies anything”. The full run (engine 1.24.0, 5 August 2026) found 12 certified documents, every genuine one declaring /P 1, no changes permitted, where the plan had predicted /P 2 would carry all the value. The prior was not slightly off; it was inverted. The same run caught the family reading a signature's own appearance image as a forbidden content change: 109 counterfactual misfires, 0 after the 1.24.0 correction, and the family has fired 0 times above info on 273 signed documents. Both write-ups stand in the repository, the first under a banner naming what it got wrong.

Even the clean results in this family of work are stated with their limits. running-balance-break fired 0 times in 3,842 published documents, and its precondition never met a real bank statement, because the corpus holds banks' published fee schedules, not customers' statements, and never will. Zero fires is not the same claim as measured-high precision. The CNJ process-number check verified 156 of 156 real numbers with 0 false accusations, and its measured recall over the 1,006 forgeries is zero, the forgeries edit amounts and dates, not case references.

What is still not measured

• Whether issuer baselines catch a clean regeneration. That is the counter we sell against the null result above, and it has not been benchmarked. It will be measured and published either way.
• A whole-population false-positive rate on the current engine. What exists is the 1.17.0 projection above plus per-family re-measurements at 1.19.0, 1.22.0 and 1.23.0. The full re-run is reproducible from the surviving manifests and has not been done.
• A true-positive rate against real fraud. Every recall figure on this page is measured against forgeries this repository built, because real fraudulent documents cannot be obtained legally at scale: a genuine forged bank statement or boleto is a stranger's personal data and usually evidence in someone else's dispute, and the corpus rules close that route by construction. No number here says how often the engine catches fraud as it actually arrives.
• Recall against a real adversary. The positives are built by a script from eight operations we chose. Nothing here says how the engine performs against a forger who reads this page.
• Image metadata is trivially strippable. exiftool -all= removes every identity and timestamp finding the image engine makes. A match is strong evidence; an absence is none, and every image finding says so in its own prose.
• The Office engine does not render the document. Text hidden by being coloured to match its background, sized to zero, or covered by a shape is not detected there. The paint-order pass that makes redaction-exposure work has no OOXML equivalent yet. It is the biggest hole in the newer medium, and it is disclosed on every Office report rather than left to be inferred.
• Encrypted PDFs are disclosed, not attacked. A password-protected file is reported as “not content-analysed” rather than scanned as ciphertext and reported as clean.

Whether a forger would hide text in a layer

A PDF can carry content in an optional-content group switched off by default: present in the file, absent from the screen. Engine 1.28.0 reads and retains those runs and judges none of them, and it established that glyph-anomalies, br-identifier-checksum and running-balance-break read such text as though it were on the page. Measured across the same 1,727 documents, and re-measured on 2026-08-07 after we found the first measurement was wrong.

The 2026-08-06 run decided "off" from the default configuration's /ON, /OFF and /BaseState lists alone. Visibility is not one flag: a configuration can also carry /AS, and a group whose /Usage /View /ViewState is /OFF is invisible on screen however the lists read. Engine 1.28.0 counted such a group as visible. The gap was reported by a reader, Tobias (@to21as), in a comment on our own redaction post, which is the honest answer to how it was found. Engine 1.29.0 reads /AS, and these are both runs, side by side.

The correction was real and it was small. One further document, and the number that carries the argument did not move: still none of 1,727 paints text into a layer that is off, and the three families still never read a hidden line. Only 5 documents (0.29%) carry /AS at all, every one of them in the US .gov sample; the Brazilian sample carries none, so its counts were never affected.

One thing the re-run found that the first run could not see, and it is the sharper case Tobias named: both US documents with a hidden layer are off for /View and on for /Print: they look redacted in the viewer and in a screenshot, and print what is hidden. That shape has no feedback loop at all. It is 2 documents in 1,727, and we would rather publish the number than the impression.

And the older limit still stands underneath all of it: these are documents nobody was hiding anything in. A null result on published files is evidence about accidents, not about adversaries.

Widened 2026-08-07 to every real document we hold. The 1,727 above is the corpus the 1.28.0 figure was measured on, and the comparison had to stay on it or a definition change and a corpus change would be indistinguishable. The standalone claim is not capped there: across ≈3,687 distinct documents (adding 1,317 Brazilian federal procurement contracts, 255 US court filings, 83 bank statements and the template sets), still none paints text into a layer that is off. 106 declare optional content (2.9%), 7 carry a group off by default (0.19%), and 7 carry /AS (0.19%). The procurement corpus contributed a clean zero on every line: not one of its 1,317 documents declares optional content at all, which is what machine report generators do, layers come from design tools.

A countersigned contract scores high, and that is the largest false positive we have measured

Run over 1,317 Brazilian federal procurement contracts published on PNCP (fetched 2026-08-07, engine 1.29.0, 0 unreadable): 89.8% low, 0.4% elevated, 9.8% high. The aggregate flatters us, it is better than both samples above, and it hides the whole story, so here is the split that matters.

On unsigned documents the engine is 99.65% low. On signed ones, 77% score high. Every one of the 37 signed documents that scored low carries exactly one revision, so the trigger is not tampering, it is a second signature.

The mechanism is not a bug, which is why it is disclosed here rather than fixed and forgotten. When a second party signs, their bytes land beyond the first party's /ByteRange, so signature-coverage reports at high that signatures no longer cover the file; each signature is an incremental revision, so incremental-updates reports at high; and /ID[1] must change per revision, so id-inconsistency reports at medium. Every statement is true of the bytes. Meanwhile signature-integrity reports info: the signatures are cryptographically intact and the signed content was never altered. The same report says the signatures are valid and scores the document 100.

A contract signed by two or more parties is the most ordinary legitimate document in this market, and the engine currently treats it as its highest-risk output. Until that is recalibrated (and it is a severity change, so it waits on the measurement above rather than preceding it), read a high band on a multi-signature PDF as uninformative. The families to read instead are signature-integrity, which is the one that actually checks the digest, and signature-permissions.

The number nobody in this market publishes

The competitive landscape was read directly on 31 July 2026: Resistant AI, Inscribe, Snappt, Ocrolus, Truv, Veryfi, DocuClipper, VerifyPDF and HTPBE. Several publish a headline accuracy figure in the 99% range with no methodology published, no stated denominator and no ground truth, which is why none of those figures is repeated here.

Not one of them publishes how often its system flags a genuine document. That absence is the reason this page exists. It is not a claim that our engine is better than theirs: several of them do things this one cannot, at a scale this one is not built for. It is a claim about what you can check. A recall-flavoured number with no denominator cannot be argued with, only believed; a false-positive rate on a named, public, uncurated population can be reproduced by anyone who fetches the same manifests.

Even the strongest public benchmark in the adjacent space leads with the false accept rate. False-reject behaviour on genuine documents is the thing every operations team actually pays for, and it is the thing nobody quotes.

There is also no independent test anyone in this market could submit to. No public benchmark exists for PDF-structure forensics: cross-reference tables, incremental updates, producer metadata, signature coverage. The academic benchmarks are pixel-level image forensics, which is a different problem. That cuts both ways, and both halves are load-bearing: nobody can prove we are worse than the field, and we cannot prove we are better. It is why this page publishes reproducible measurements of our own error, rather than a ranking no one can run.

Sources for every figure on this page are tracked in this product's own repository: the engine release notes in CHANGELOG.md (1.17.0, 1.18.0, 1.19.0, 1.22.0, 1.23.0, 1.27.0), the “Honest gaps” section of STATUS.md, the committed phase-0 aggregates in docs/CORPUS-PHASE0-*.md and docs/GLYPH-PHASE0-2026-08-05.md for the failed measurements, and the evidence compilation in docs/EVIDENCE.md §5.2, §5.3 for the competitive reading above. Per-document corpus findings are not published: the corpus rules delete the documents after a run, and publishing a list of published organisations' filenames beside a risk band is not something a measurement needs.