Authors Guild test finds some AI detectors perfectly identify human writing while others fail on every single text

2026-06-25 · Source: The Decoder · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Publishing & Journalism · Depth: Novice, short

Summary

The Authors Guild conducted a test of AI text detectors, revealing significant performance disparities among tools. Pangram and Grammarly accurately identified all ten human-written articles, published between 2020 and 2022, as human-generated. Originality.ai also performed well, showing zero percent AI detection for most texts. Conversely, Sidekicker.ai flagged every article as mostly AI-generated, with two scoring 100 percent, while ZeroGPT also proved unreliable, reporting high AI percentages for human content. The Guild warns that even top-performing tools should not be the sole basis for decisions, as false positives can jeopardize authors' contracts and reputations. This highlights a "troubling paradox" where highly crafted human writing can statistically resemble AI output.

Key takeaway

For publishers evaluating manuscripts or platforms assessing content originality, you should exercise extreme caution with AI detection tools. Relying solely on these tools risks unjustly penalizing human authors, potentially costing them contracts and damaging reputations due to false positives. Always disclose your detection methods and provide authors a clear process to appeal or defend their work, recognizing that even well-crafted human text can be misidentified.

Key insights

AI text detectors vary widely in accuracy, with some reliably identifying human text while others frequently produce false positives.

Principles

• Highly crafted human writing can statistically resemble AI output.
• Detector accuracy is not guaranteed and tools change constantly.
• Minimizing false positives for human text is a key tuning goal.

In practice

• Publishers should disclose AI detection methods used.
• Authors need opportunities to defend against AI detection flags.
• Avoid using single AI detector results for critical decisions.

Topics

• AI Text Detection
• False Positives
• Authors Guild
• Content Originality
• Publishing Ethics
• Language Models

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Legal Professional, Tech Journalist

Related on AIssential

• When is detecting AI-generated text worthwhile? — AI text detection's value depends on explicit assumptions about author intent, population, and the costs of detection errors.
• How to Humanize AI Text: What We Learned from 19,804 Text Pairs — Targeted AI text rewriting can substantially lower detector scores, but its effectiveness varies by topic and measurement.
• Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews — AI-generated peer reviews converge semantically, while human reviews, even LLM-refined, retain unique ideas.
• Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection — AI-text detectability is non-monotonic, influenced by edit operations and revision history, not just AI content proportion.
• Do we really need to detect LLM-generated text? — Masked word prediction by LLMs can distinguish AI-generated text from human text without explicit training.
• We Asked the ‘Future of Truth’ Author to Explain How He Used AI. It Didn’t Go Well — The irony of a book on AI and truth containing AI-generated falsehoods highlights critical authorial responsibility.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Decoder.