Are AI Detectors Accurate?

The short answer

• Where they do well. Unedited output from the models they were trained on, set against ordinary published human writing, at a reasonable length. In a lab test on a public benchmark of older generators, an open-source detector separated the two almost completely (an AUROC of 0.9988, where 1.0 is perfect): a lab result, not a promise about real use.
• Where they don’t. Text that has been edited, paraphrased or mixed with human writing; short text; text from newer generators; and writing by people whose English is plain or non-native.
• What that means. A detector’s answer is a reason to look closer, not a finding about a person. The rest of this guide shows the evidence and how to read a claim.

The number that matters: false positives

Accuracy, the share of texts a detector gets right, hides the mistake that hurts most. On a test with as many machine texts as human ones, a tool can score well while still wrongly flagging honest writers, because the two kinds of mistake are counted together. Ask for the false-positive rate instead: of the texts people wrote, how many does it call AI?

Small rates add up. When Vanderbilt University explained in August 2023 why it was disabling Turnitin’s AI detector, it worked from Turnitin’s claim of a 1% false-positive rate: the university had submitted 75,000 papers to Turnitin in 2022, so about 750 could have been wrongly labeled as partly written by AI (Vanderbilt). The rate also depends on who is writing, as the next section shows.

What independent tests and regulators found

These are different tests of different tools at different dates, so they do not add up to one number. What they share is the pattern: claims of very high accuracy on friendly test data, and much weaker results once the writing has been edited, or comes from a group the tool was not built for, or is not the kind of text the tool was tested on.

What we measured

We ran an open-source detector, a RoBERTa model fine-tuned on human and machine text, through our own pipeline on 2,656 texts from RAID: 1,189 human texts and 1,467 machine texts, 332 of them edited by one of RAID’s attacks. It was a test on older generators and no non-native writers, so read it as a lower bar rather than a promise, and not as an accuracy claim for this site.

Then we edited the machine texts, as a student trying to slip past a detector might. Each row applied one of RAID’s edits to 29 to 40 machine texts and shows how many of them were still called “Likely AI” and how many now read as “Likely human”. Some of these texts are under 250 words, which cannot be called “Likely AI” at all, so the first figure understates what the score alone would say.

After six of the nine edits, fewer than half of the machine texts were still called “Likely AI”. The figure that should worry a reader most is the last one in the first row: with nothing but paragraph breaks added, 7 of 29 machine texts, nearly a quarter, read as human. The samples are small (29 to 40 texts each), so treat the rows as a warning, not a measurement.

What this test cannot show: RAID’s machine texts come from six generators of four families released between 2019 and 2023, not today’s chat models; it holds no writing by non-native speakers, the group the studies above found most at risk; its human texts are published news, reviews, books, recipes, poetry, forum posts, abstracts and Wikipedia, not the essays and emails people actually check; and the cut-offs we used were provisional. We publish a full report only for a detector that has passed an evaluation of its own.

Why advertised numbers and real results differ

• Friendly test data. A test set drawn from the same sources as the training data flatters a detector. Fresh texts, from other genres and newer models, are harder.
• Accuracy on balanced sets. With equal numbers of human and machine texts, one headline number can hide a false-positive rate that matters a great deal to the people flagged.
• A cut-off chosen after seeing the test. Cut-offs tuned on the test set inflate the result. They should be set on one set and measured on another.
• Whole documents against short passages. Detection is easier over thousands of words than over a paragraph, and a claim that doesn’t say which is hard to compare.
• Unedited machine text. Real use includes text a person has revised, translated, polished with an assistant or run through a paraphraser.
• Yesterday’s generators. A detector recognizes what it was trained on. Turnitin lists the models its English detector can detect and says it will keep expanding the list, which is the same point from the other side.

Questions to ask of any accuracy claim

1. What is the false-positive rate, and at what cut-off?
2. Was it measured on writing like this: the same genre, length and kind of writer, including non-native English?
3. Which generators were used, and how old are they?
4. Was the machine text edited, paraphrased or mixed with human writing?
5. Who ran the test, and can anyone see the method, the sample size and the date?
6. Does the tool say “uncertain” when the evidence is thin, or does it always pick a side?
7. Which version of the model produced the result?

If you use a detector, or one has flagged you

• If you are checking someone’s writing: use the result to start a conversation, and look for other evidence before drawing a conclusion. Turnitin’s own advice is the same: no score is a “right” one, and highlighted text is a prompt for a conversation, not a conclusion.
• If your writing was flagged: ask what the score measures and what it was tested on, and keep drafts and version history that show how the work developed. The guide to the Turnitin AI detector explains what that tool’s report does and doesn’t say.
• If you want a second opinion: different detectors use different models and will disagree. Treat two tools that agree as slightly stronger evidence, and two that disagree as a sign that the evidence is thin.

Sources

• Weber-Wulff and colleagues (2023): Testing of detection tools for AI-generated text. A test of 12 public tools and two commercial systems.
• Liang and colleagues (2023): GPT detectors are biased against non-native English writers. False-positive rates for TOEFL essays versus US eighth-grade essays.
• Dugan and colleagues (2024): RAID. The benchmark behind our test, and its finding on detector robustness.
• Krishna and colleagues (2023): Paraphrasing evades detectors of AI-generated text. The effect of paraphrasing on DetectGPT, watermarks and commercial detectors.
• OpenAI: New AI classifier for indicating AI-written text. The launch figures for OpenAI’s classifier and the note that it was withdrawn.
• FTC: FTC Order Requires Workado to Back Up Artificial Intelligence Detection Claims (April 2025). The 98 percent claim and the 53 percent test result.
• FTC: Final order against Workado (August 2025). The finalised order on AI detection accuracy claims.
• Vanderbilt University: Guidance on AI Detection and Why We’re Disabling Turnitin’s AI Detector. The 75,000-paper arithmetic and the university’s reasons.
• Turnitin: AI writing detection capabilities FAQs. Turnitin’s false-positive claim, its test set, and its guidance on using the score.
• Turnitin: Understanding the false positive rate for sentences. Turnitin’s advice on using its indicator as a prompt for a conversation.

Facts and figures were checked against these sources on September 30, 2026. If something is out of date or wrong, tell us.