[R] ICLR: Guess which peer review is human or AI?

2026-02-11 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Intermediate, quick

Summary

A game hosted on reviewer3.com challenges participants to distinguish between human-written and AI-generated peer reviews from ICLR. The game, which presents two reviews for a given paper, has led many players to suspect it is a data collection effort by "reviewer3" to evaluate their AI models. Participants quickly identified several heuristics for detecting AI-generated reviews, including shorter text length, simpler wording, and a lack of formatting like bolding, italics, or LaTeX equations. Many players reported achieving near-perfect scores by consistently assigning the shortest or least formatted text to human reviewers. The consensus among players is that current LLM-generated reviews are often insubstantial, repetitive, and ultimately time-wasting.

Key takeaway

For AI Scientists evaluating the utility of LLMs in academic peer review, this game highlights current limitations. Your models likely produce reviews that are easily identifiable as AI-generated due to their brevity, lack of formatting, and superficial content. Focus on developing prompts that encourage deeper analysis, specific feedback, and the inclusion of standard academic formatting to improve AI review quality and reduce detection.

Key insights

AI-generated peer reviews are often detectable by their brevity, simpler language, and lack of formatting.

Principles

• Shorter text often indicates human authorship in reviews.
• AI output quality correlates with prompt quality.

In practice

• Prioritize shorter, less formatted reviews as potentially human.
• Craft detailed prompts for higher quality AI text generation.

Topics

• AI Peer Review
• AI Content Detection
• Large Language Models
• ICLR Reviews
• AI Writing Quality

Best for: AI Scientist, AI Researcher, Research Scientist, Prompt Engineer

Related on AIssential

• PeerPrism: Peer Evaluation Expertise vs Review-writing AI — Current LLM detectors conflate text generation with intellectual contribution in hybrid human-AI peer review.
• Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions — AI review can significantly align with human feedback and identify novel issues in paper drafting.
• The AI Scientist: Towards Fully Automated AI Research, Now Published in Nature — AI agents can now autonomously conduct machine learning research, generate papers, and pass human peer review.
• [D] ICML paper to review is fully AI generated — AI-generated research papers challenge conference policies and reviewer assessment of technical merit and authorship.
• Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI — LLMs are making peer reviews longer and more fluent but reducing focus on deep evaluative dimensions.
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes — An LLM-based system can accurately predict peer-review outcomes and provide structured feedback for academic manuscripts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.