Reference: API quickstart
The uploads endpoint of a fraud checker is an invitation addressed to hostile files. Ours takes anonymous uploads, on purpose. So every parser behind it runs on input chosen by whoever is most motivated to hurt it. Engine 1.13.1 exists because three string scans in that path were quadratic: double the input, and the work goes up fourfold. The release that fixed them is the closest thing this changelog has to a security bulletin about ourselves.
Three scans, three numbers
| input | where | before | after |
|---|---|---|---|
| a 6 KB PDF | the XMP picker read the untruncated packet | 92.5 s | 5 ms |
| a 21 KB docx | docProps/custom.xml hand-rolled a regex reader | 444 s | 68 ms |
400 KB of < | stripTags, reachable from every Office read | 71 s | 0 ms |
The third one deserves its footnote. stripTags decides what
every author name, title and tracked-change body reads as. So I checked the
rewrite against the regex it replaced over 200,000 fuzzed inputs, and the
suite keeps the two behaviours a cleaner rewrite would have dropped. It is
not quote-aware, and an unterminated < stays literal text.
All three fixes are pinned by wall-clock tests. A performance property nobody asserts is a performance property somebody will quietly trade away.
Why the parsers are hand-written and rude
The same reasoning shows up everywhere in this engine's architecture. The ZIP reader, the XML reader, the JPEG reader and the DER reader are all hand-written, small, and famous mostly for what they refuse. DER is the binary format signatures are written in.
A general-purpose library is a large, capable parser. On attacker-controlled input, capability is cost: every feature it supports is a branch an attacker can spend your CPU in.
The house versions read exactly what a signal needs. They return null at the first sign of a structure that wants to be expensive: an XML entity that resolves, an indefinite-length DER node, a declared length past the buffer.
Where refusal is not possible, there are budgets, and the budgets obey two rules learned the hard way. First, a budget is document-wide, never per-item, because a per-stream cap bounds nothing when the attacker chooses the number of streams. This repository has learned that lesson at least three times. The most recent was when the content walker started entering Form XObjects, and its operator ceiling had to become global on the spot.
Second, a budget that runs out is disclosed in the report. A truncated page and a quiet page must never look alike, so the retention caps keep their tallies and the walker names the forms it declined to enter. A cap the caller cannot see is a false negative wearing an optimisation's clothes.
Since 1.34.0, the cost is an output
The parser has always spent decompression against a per-document budget. What changed in 1.34.0 is that it keeps the receipt: every report, for PDF, image and Office alike, carries a cost block.
On every report
summary.cost = { physicalBytes, expandedBytes, expansionRatio }
Bytes read, bytes of decompressed output produced, and the ratio between
them. Both counts are deterministic, so they ride inside the byte-identical
report body. Wall-clock time is not deterministic, so it stays out of the
default body and appears only behind an explicit ?timing=1.
For an agent doing document intake, the ratio is the useful number. A file that expands to hundreds of times its physical size has announced its intentions before any signal fires. The cheap triage mode exists so that announcement can be read without paying for the full parse.
The service wraps the engine in its own limits. A 10 MB upload cap, checked before the transfer costs anyone a round trip. Inspections on worker threads rather than the main event loop, so one slow file cannot stall the others. And a timeout with queue shedding, exercised against the compiled worker in tests rather than assumed.
The client got its half too. A review row showed a 40 MB file uploading in full just to receive the server's 413, its "too large" answer. The gate now runs before the first byte leaves the browser.
The title, argued once
Every one of these numbers is a decision about who pays for a hostile file. Either the uploader pays, in a refusal or a disclosed truncation, or every other user pays, in a saturated server.
A parser without stated limits has still made that decision, and it made it in the attacker's favour. It will publish the decision as an outage instead of a changelog entry.
Writing the caps down, testing them by the clock and printing the cost on the report: that is the same policy this product applies to accuracy claims, pointed at ourselves.
The API quickstart shows the anonymous call: one curl, no
key, ten documents an hour. The summary.cost block is on every
response, including the free ones. You can price a document's weight before
you trust it.