The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use

2026-06-04 · Source: cs.SE updates on arXiv.org · Field: Science & Research — Research Methodology & Innovation, Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Advanced, extended

Summary

A recent questionnaire survey involving 120 Empirical Software Engineering (ESE) community members, predominantly seasoned academics from Europe and North America, reveals significant challenges in the peer review system. Two-thirds of respondents perceive their review load as high or very high, averaging 25-32 reviews annually, with conferences accounting for two-thirds of this effort. While participants generally rate their own review quality highly, they frequently cite high workload (110 respondents), mismatched expertise (64), and insufficient recognition (51) as obstacles to quality. Common issues in received reviews include shallowness (82), generic feedback (59), and unrealistic demands (48). Over half of respondents (70) do not use LLMs for reviewing, but among those who do, public services like ChatGPT (36) are common, often for presentation or politeness. The community is divided on exploring LLM integration, but a strong majority (81) supports banning unethical LLM use by authors and reviewers. Suggestions for improvement focus on reducing review load, enhancing governance, and responsibly integrating LLMs.

Key takeaway

For Research Scientists and Directors of AI/ML managing publication strategies, recognize that the ESE peer review system is strained by high workloads and inconsistent quality. You should prioritize ethical GenAI use, providing clear guidelines and enforcing strict penalties for misuse. Advocate for improved reviewer incentives and consider implementing early desk rejections to alleviate reviewer burden and enhance overall review quality within your community.

Key insights

The ESE peer review system faces unsustainable workload, quality issues, and ethical dilemmas exacerbated by GenAI, demanding urgent systemic changes.

Principles

• High workload and low incentives degrade review quality.
• Mismatched reviewer expertise impedes effective evaluation.
• Unethical GenAI use in reviews requires strict penalties.

Method

Conducted a questionnaire survey with 120 ESE reviewers, using 22 questions (mostly multiple-choice) covering load, quality, LLM use, and improvement suggestions, with pilot testing and anonymous data collection.

In practice

• Improve reviewer incentives and recognition.
• Implement early desk rejection for submissions.
• Provide clear guidelines for responsible LLM use.

Topics

• Peer Review
• Empirical Software Engineering
• Generative AI
• Large Language Models
• Research Ethics
• Academic Publishing

Best for: Research Scientist, AI Scientist, Director of AI/ML

Related on AIssential

• The peer review system is breaking down. Here’s how we can fix it — The voluntary peer review system is failing due to unacknowledged labor and increased academic pressures.
• Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI — LLMs are making peer reviews longer and more fluent but reducing focus on deep evaluative dimensions.
• Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions — AI review can significantly align with human feedback and identify novel issues in paper drafting.
• Backlash against Arxiv's proposed 1 year ban is genuinely perplexing. [D] — ArXiv's proposed ban on AI-generated "slop" reveals deep-seated issues in academic publishing ethics and accountability.
• 303 Plagiarism Retractions Since 2020 - Is India’s Research System Broken? — India's research system faces a severe credibility crisis driven by ranking incentives and insufficient quality control.
• Why AI can’t be trusted to write scientific reviews — Current AI tools are not ready for high-stakes scientific systematic reviews due to limitations in context, nuance, and transparency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.