How to detect AI-generated text

Three families of method are sold as an answer to this question, and only one of them is deterministic. Which one applies to you depends on whether you are holding bare text or a file.

Tamperlens does not detect AI-written text, and this page is not a soft sell for a feature that is coming. It exists because the question is almost always asked while holding a document, and a document is checkable in ways a paragraph never is.

The three approaches, and what each can establish

Approach Deterministic or probabilistic What a positive result establishes What it can never establish
Commercial AI-text classifiers Probabilistic, and poorly calibrated outside the data they were built on That the prose resembles instruction-tuned model output Who or what wrote it. There is no authorship measurement inside the score
Statistical watermarks (SynthID-Text, and Claude's, which is a version of it) Deterministic-ish: a keyed statistical test with a genuine p-value, but only for whoever holds the key That a specific vendor's keyed model was probably involved at some point That the model wrote rather than edited it; that a human did not; and absence proves nothing at all
File-level checks, when the text lives in a document Deterministic: the file either contains the structure or it does not What the file's own bytes record: revisions, producing tool, signature coverage, fonts, dates, arithmetic, signed provenance Whether a language model composed the sentences. It does not look at prose at all

Read down the last column and the shape of the problem appears. The two methods aimed at the writing cannot answer the question, and the method that can return facts is aimed somewhere else. The move is not a better detector. It is to notice you are holding a file, and ask the file.

1. Commercial classifiers: probabilistic, and broken in a specific way

What they measure is the stylistic residue of instruction tuning, not machine authorship. This has been shown mechanistically rather than merely alleged: feed the same model family through detectors and the base model's text reads as overwhelmingly human while its instruction-tuned sibling gets flagged.

“Current detectors are tracking artifacts of instruction tuning and local context more than any invariant notion of machine-generated text.” A detector that learned “sounds like a well-behaved assistant” flags anything else that sounds that way, and a great deal of honest human writing does.
same weights · same size · same corpus base model detector: “human” instruction-tuned sibling detector: “AI” The only variable is the fine-tune. That is what the score is tracking.

The false positives, with the numbers

Formulaic writing reads as machine writing, and so does careful writing in a second language.
FALSE-POSITIVE RATE ON TEXT KNOWN TO BE HUMAN-WRITTEN granted patent claims 78.3% non-native English writers 61.22% flagged by all seven at once 19.78% native-speaker essays ≈5% Every bar is text a human demonstrably wrote. The axis is how often a detector called it AI. Patent claims and second-language prose are both low-perplexity, which is the axis being measured.
  • Non-native English writers: 61.22%. Seven commercial detectors on human-written TOEFL essays, 19.78% flagged by all seven unanimously; the same detectors on native-writer essays, around 5%. Liang et al., Patterns (arXiv 2304.02819). A prompt asking for more sophisticated language both removes the bias and evades the detector, which tells you the axis being measured.
  • Granted patents: 78.3% at the claim level. On 500 granted European patents (human-written by definition, pre-dating the models), a leading open detector called four claims in five machine-generated (arXiv 2607.13044). Patent law's clarity requirements manufacture low-perplexity human prose.
  • Human text with an LLM pass over it: the realistic case, and the worst one. A detector holding 93.5% true positives at a strict 1% false-positive setting falls to 15.1% on human text an LLM was asked to revise (arXiv 2607.29539). In the “who wrote which part” setting, the best system at PAN 2025 reached 65.06 F1.
  • Every tool below 80% in the definitive academic evaluation. Weber-Wulff et al., fourteen tools across six document classes (International Journal for Educational Integrity 19(1):26). Accuracy on human-edited AI text: 30%. On paraphrased AI text: 15%. Verbatim conclusion: “the systems we tested should not be used in academic settings.”

The extrapolation that matters for commercial documents. Invoices, contracts and KYC paperwork are more formulaic and lower-entropy than a patent claim, and are usually short and frequently not in English: the two other regimes where every method degrades toward chance. Expect a 60-85% false-positive band. That figure is an extrapolation from the patent and non-native-speaker results, not a measurement of business documents, and we say so rather than dress it up.

What the vendors did when the numbers arrived

“We tested our model in a controlled lab setting… Since our release, we discovered real-world use is yielding different results from our lab.”
THE VENDOR'S OWN RETREAT, IN ORDER launch On by default for more than ten thousand institutions +6 weeks The chief product officer's update minimum length raised · launch false-positive claim rescoped to documents >20% AI 2024 Scores between 1% and 19% suppressed entirely 2026-08 “AI-generated” vs “AI-modified” abolished, “to reduce the risk of overinterpretation” The sentence-level false-positive rate it disclosed is no longer published anywhere.

Quotation and the update from Turnitin's chief product officer. Institutions that switched it off, individually verified: Vanderbilt, Washington State, Waterloo and around two dozen more. None has re-enabled it.

That update raised the minimum document length, retroactively rescoped the launch false-positive claim to documents more than 20% AI, and disclosed a sentence-level false-positive rate: a tenth of which sat nowhere near any AI writing. Sentence highlights are what instructors accuse students on, and that figure is no longer published anywhere.

Since 2024 scores between 1% and 19% are suppressed entirely, and in August 2026 the distinction between “AI-generated” and “AI-modified” was abolished outright, “to reduce the risk of overinterpretation.”

  • ~750papers a year wrongly flagged: Vanderbilt's own arithmetic before it switched the detector off
  • ⅓of Washington State's AI integrity hearings ended in “not responsible”
  • 100%AI: what Waterloo's internal test scored on human text
  • 0institutions have re-enabled it

The sources: Vanderbilt (75,000 papers a year × the vendor's own claimed 1%), Washington State (February 2026), Waterloo. Around two dozen universities have been individually verified as having switched it off, and Inside Higher Ed counts “at least a dozen” naming names.

The perverse result

Detectors flag disclosed, honest assistance and miss deliberate evasion. One 2026 study measured honestly-disclosed light refinement flagged at 64-80%, against under 4% for humanizer-processed text (arXiv 2608.11256). Grammarly's rewrite function has been shown to turn wholly human text into a full AI flag. The student who declares their use is at greater risk than the one who conceals it. The exact inversion of what the tool is bought to do.

And the cost is not evenly distributed. In a nationally representative survey, 10% of US teenagers reported being falsely accused of using AI: 20% among Black teenagers, 7% among white. Read with the 61.22% figure above, that is a tool whose errors fall on the people least able to contest them.

The student who declares their use is at greater risk than the one who conceals it: the exact inversion of what the tool is bought to do.
HOW OFTEN A DETECTOR FIRES honestly disclosed light refinement 64-80% the same text run through a humanizer under 4% The detector fires on the disclosure, not on the concealment. AND THE ERRORS ARE NOT EVENLY DISTRIBUTED 10% of US teenagers report being falsely accused of using AI. 20% among Black teenagers · 7% among white teenagers.

Flagging rates from arXiv 2608.11256 (2026); the false-accusation figures from a nationally representative survey.

Verdict: a classifier score is never evidence. At its very best it is a weak prior telling you to go look at something checkable.

The single most informative fact in the category

On 31 January 2023 OpenAI shipped an AI Text Classifier and published its measured performance: it “correctly identifies 26% of AI-written text (true positives)… while incorrectly labeling human-written text as AI-written 9% of the time.” The launch notes said plainly: “It should not be used as a primary decision-making tool.” On 20 July 2023 OpenAI withdrew it “due to its low rate of accuracy.”

Three years later, the organisation with the most training data and the strongest incentive to sell text detection ships watermarking for images and audio and C2PA on files, and nothing for text. When the vendor best placed to build it measured it, killed it and never returned, that is stronger than any benchmark.

The organisation with the most training data and the strongest incentive to sell text detection built one, measured it, withdrew it, and has never returned.
2023-01-31 AI Text Classifier ships 26% of AI-written text identified 9% of human text wrongly labelled 2023-07-20 Withdrawn “due to its low rate of accuracy” never replaced 2026 Watermarking for images and audio · C2PA on files Nothing for text. When the vendor best placed to build it measured it and stopped, that is stronger than any benchmark.

2. Statistical watermarks: a real test, gated behind someone else's key

This is the one genuinely sound idea in the category, and it is essentially unavailable to you. As the model writes it hashes the last few tokens with a secret key, and uses that seed to bias which of several equally good next words it emits. In unmarked text those tokens arrive at chance rate, so counting them gives a z-score and a genuine p-value. That is the structural advantage over a classifier: a calibrated false-positive rate instead of a vibe.

There is one scheme here, not two. Anthropic's own documentation states that “Claude's text watermark is a version of the SynthID-Text approach published by Google DeepMind”, so the Nature 634, 818-823 paper is the specification for both: most coverage of the August 2026 announcement called the mechanism undisclosed. Claude models launched on or after 2 August 2026 watermark at generation, worldwide, driven by Article 50(2) of the EU AI Act.

The gate: detection needs the key

The scheme is symmetric: without the secret key you cannot reconstruct the seeds, and without the seeds there is nothing to count.
model + secret key biases the sampling loop watermarked text looks identical to a reader you, no key no seeds, nothing to count the test cannot be run at all the vendor: holds the key z-score and a real p-value the algorithm is open; the key is not SynthID text detector: promised “in the coming weeks”, May 2025. Still image, video and audio only. Anthropic detection API: announced. No endpoint, no published error rates, no threshold, no date.

So today, for a third party holding a suspicious paragraph, the answer is not “run the watermark test and see”: the test cannot be run at all.

What a hit would prove, and what it never could

Involvement, not authorship. Anthropic states the semantics precisely: a watermark “can only determine that Claude was likely involved with the content at some point,” and it “cannot distinguish ‘Claude wrote this’ from ‘Claude heavily edited this.’” It says nothing about who supplied the substance, the facts, or the intent.

“Watermarking is sparser on factual passages where there are fewer choices that can be made without decreasing the accuracy of the text.”
THE MARK NEEDS SOMEWHERE TO HIDE free prose an essay, an explanation, a story many equally good next words the mark rides along document-shaped text totals, dates, account numbers, legal boilerplate, form fields the mark thins out AND BEFORE ANY ADVERSARY APPEARS A state-of-the-art distortion-free scheme detects about a quarter of roughly hundred-token responses at p ≤ 0.01. Insert a character between every word and delete it: unchanged to a reader, re-tokenised for the detector, mark gone. For under $50 an attacker can both spoof and scrub schemes previously considered safe, succeeding over 80% of the time.

Everything below removes or fakes the mark:

  • Short passages carry too little signal. Even a state-of-the-art distortion-free scheme detects only about a quarter of roughly hundred-token responses at p ≤ 0.01, before any adversary appears.
  • Re-tokenisation destroys it, for free. Insert a character between every word and then delete it: the string is unchanged to a reader and re-tokenised for the detector, which destroys the n-gram contexts that seed every hash-based scheme. OpenAI names this attack in its own writing on why it never shipped text watermarking.
  • Paraphrase and translation degrade or remove it. A paraphrasing model removes it outright below roughly 800 tokens, and an open-weights model never carried one: the watermark lives in the sampling loop, not the weights.
  • It can be stolen, scrubbed and forged. Querying a watermarked API approximates the key material: “for under $50 an attacker can both spoof and scrub state-of-the-art schemes previously considered safe, with average success rate of over 80%” (Jovanović, Staab and Vechev, ICML 2024). Spoofing is the direction nobody plans for. A companion result shows two weak oracles suffice to remove marks from all three major schemes (Zhang et al., ICML 2024).

A mark that can be forged for under $50 cannot support an adverse finding against a person. Not a disciplinary outcome, not a dismissal, not a fraud allegation. Watermarking is a real and useful speed bump against low-effort misuse at platform scale. It is not an evidentiary instrument, and it was never built to be one.

3. When the text lives in a file, the ground turns solid

Bare text has no container, which is why C2PA has no text asset type at all. A paragraph is a byte sequence, and copy-paste discards everything except the characters. There is nowhere in a paragraph to put a signature that survives being selected and copied.

A document is different. A PDF is a container with a history, and the history is not a guess:

A paragraph has nowhere to keep a receipt. A file keeps five, and none of them is about the prose.
a paragraph pasted into a box no container · nothing survives copy-paste the same words inside a file a container with a history WHAT THE FILE RECORDS, DETERMINISTICALLY Revisions editors save by appending: what changed, and when Producing toolchain producer and creator strings name the software Metadata agreeing with itself Info dict vs XMP, the timestamps, the /ID pair Signature coverage whether a signature still covers what it signed Printed arithmetic totals that do not add, check digits that do not recompute None of this answers “did a language model write these sentences”, and none of it pretends to.
  • Revisions. PDF editors save by appending, so the revision history shows what changed, see incremental update vs full rewrite.
  • The producing toolchain. Producer and creator strings name the software that wrote the bytes, a creating tool is a different fact from a consuming one.
  • Metadata that has to agree with itself. The Info dictionary and the XMP packet, creation and modification stamps, the trailer /ID pair: see PDF metadata forensics.
  • Signature coverage. Whether a signature still covers what it signed is arithmetic, not opinion, signature signals.
  • Printed arithmetic. Totals that do not add, check digits that do not recompute, number and structure signals. This survives every rewrite, because it is a property of the claims, not the container.
  • Signed provenance, where it exists. C2PA manifests can be embedded in PDF 1.7 and 2.0, and verifying one is a signature check rather than a guess: essentially zero false positives, unforgeable without the signing key, and. Unlike a text watermark, anyone can verify it.

None of that answers “did a language model write these sentences,” and none of it pretends to. It answers questions that hold up under challenge: written once or rewritten; producing tool matching what the document claims to be; numbers agreeing with each other. The companion guide, detecting an AI-generated PDF, works through the document-shaped version.

And it degrades honestly. No manifest means “no claim”, not “written by a human”. A quiet report means the file did not record anything, not that nothing happened.

What none of this can tell you

There is no test for human authorship. Every method above is either a resemblance guess or a check for a mark a machine left behind. Neither has a negative form that means “a person wrote this.” A clean classifier score and an absent watermark are the same non-answer.

  • Absence of a watermark means nothing. Open-weights models never emit one. Older versions are still in transition periods. Paraphrase, translation, a character insert-delete pass or simply retyping removes it.
  • A classifier score is not evidence against a person, and acting on one is discriminatory in practice. These tools misfire on second-language writers at roughly twelve times the rate they misfire on native writers, and the false-accusation burden falls hardest on multilingual, disabled and neurodivergent people. Courts have started to say so, and the liability landed on the institutions that used the tool, not the vendors that sold it.
  • A watermark hit is not authorship either. It is involvement, by the vendor's own definition, and it is forgeable for under $50.
  • AI-written is not the same as fraudulent. Drafting a proposal or a report with an assistant is ordinary, and increasingly it is what the writing software does anyway. Fraud is a question about facts: whether the payment happened, whether the totals add up, whether the file was altered after it was issued.
  • Tamperlens does not score text. Upload a document and you get structural signals about the file, with the evidence behind each one and its benign causes. You will never get a number claiming a machine wrote the words, because we cannot produce one honestly.

So what should you actually do

If you are holding a file, check the file. The free checker reports what the structure records (revisions appended after creation, the producing toolchain, metadata that disagrees with itself, signature coverage, arithmetic that does not hold), each with its evidence and its innocent explanation.

If you are holding bare text, no tool will settle it. Ask for the source file and check that instead. Verify the claims against whoever issued them. If the concern is process rather than fraud, the answerable questions are about drafts, sources and the ability to discuss the work, not about a percentage.

Ask the file instead

Open the “Signed, then modified” sample: a precomputed report on a document whose signature no longer covers the bytes underneath it. It is the first row of the table above with the evidence attached, and it is a fact about the file rather than a guess about the prose. No account, no upload. Your own file goes into the same checker: the file is sent over HTTPS, parsed in memory and never written to disk.

Tamperlens reports risk signals, not authenticity verdicts, and it will never return a number claiming a machine wrote the words.

Related