Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

2025-12-01 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Expert, extended

Summary

Sem-Detect is a novel authorship detection method designed for peer reviews, distinguishing between fully human-written, LLM-refined human, and entirely AI-generated content. It operationalizes the principle that authorship should consider not just textual features but also the semantic content of ideas, judgments, and claims. The system compares a target review against multiple AI-generated reviews of the same paper, utilizing the observation that AI models tend to converge on similar points, whereas human reviewers introduce more unique and diverse insights. Evaluated on a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in binary detection. In the three-class scenario, it misclassifies fewer than 3.5% of LLM-refined human reviews as AI-generated, demonstrating robustness to unseen models and cross-domain transfer.

Key takeaway

For research integrity officers or journal editors evaluating peer review authenticity, Sem-Detect offers a robust framework to differentiate between genuinely human feedback, LLM-polished human reviews, and fully AI-generated submissions. This distinction is crucial for upholding scientific standards without penalizing legitimate AI assistance. You should consider integrating claim-level semantic analysis into your detection protocols to improve accuracy and reduce false accusations, especially given the method's proven robustness across diverse generation conditions and domains.

Key insights

AI-generated peer reviews converge semantically, while human reviews, even LLM-refined, retain unique ideas.

Principles

• Authorship detection should combine textual and semantic features.
• AI models reviewing the same paper converge on similar claims.
• LLM refinement preserves human semantic signals.

Method

Sem-Detect extracts textual features and claim-level semantic features by comparing a target review to multiple AI-generated references of the same paper. These features train a LightGBM classifier.

In practice

• Use claim-level semantic analysis for robust AI text detection.
• Consider confidence thresholding for deployment to reduce false positives.
• Utilize the released 20,000+ peer review dataset for research.

Topics

• AI Text Detection
• Peer Review Integrity
• Large Language Models
• Semantic Analysis
• Claim-Level Detection
• Machine Learning Conferences

Code references

• avduarte333/Sem-Detect

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• PeerPrism: Peer Evaluation Expertise vs Review-writing AI — Current LLM detectors conflate text generation with intellectual contribution in hybrid human-AI peer review.
• Do we really need to detect LLM-generated text? — Masked word prediction by LLMs can distinguish AI-generated text from human text without explicit training.
• Authors Guild test finds some AI detectors perfectly identify human writing while others fail on every single text — AI text detectors vary widely in accuracy, with some reliably identifying human text while others frequently produce false positives.
• Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest — Modern LLMs demonstrate varied capabilities across social media analytics tasks, with GPT-4 excelling in authorship verification and Gemini in attribute inference.
• AI system spots fake reviews by combining text, images and user behavior — Multimodal AI combining text, images, and behavior significantly improves fake e-commerce review detection.
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews — AI review utility stems from textual justification, not just scalar scores, requiring text-centric evaluation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.