[D] First time reviewer. I got assigned 9 papers. I'm so nervous. What if I mess up. Any advice?

2026-02-26 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

A first-time reviewer, assigned nine papers for an A* conference, expresses significant nervousness and imposter syndrome, seeking advice on managing the workload and ensuring review quality. Experienced reviewers recommend immediately contacting the area chair to request a reduced load, ideally aiming for 3-4 papers, especially given the high number and lack of prior experience. They advise against using AI for direct review, citing typical conference policies that only permit AI for grammar/spelling, with severe penalties for violations. The discussion highlights that many submissions are low quality, with some papers being AI-generated, off-topic, or missing PDFs, even at top-tier conferences. Reviewers suggest focusing on identifying obvious flaws, summarizing strengths and weaknesses, and being factual, while also emphasizing the importance of understanding conference-specific expectations and review templates.

Key takeaway

For a Research Scientist or AI Student undertaking their first academic paper reviews, immediately contact your area chair to request a reduced paper load, ideally to 3-4 papers, explaining your lack of prior experience. Strictly avoid using AI for content review, as this can lead to severe penalties; limit its use to grammar checks if permitted. Focus on factual assessment of the paper's strengths and weaknesses, and be prepared to adjust your stance during the rebuttal phase, especially with low confidence scores.

Key insights

First-time academic reviewers should manage workload proactively and adhere strictly to AI usage policies.

Principles

• Prioritize workload management by communicating with area chairs.
• Adhere to conference AI policies, typically restricting use to grammar.
• Focus on factual assessment of paper strengths and weaknesses.

Method

For new reviewers, identify obviously flawed papers first. Summarize good and bad points, and list questions. Set low confidence and be flexible during rebuttal. Consult review guides and old conference reviews.

In practice

• Contact area chair to reduce review load to 3-4 papers.
• Review old conference papers to understand quality expectations.
• Use AI only for grammar/spelling, never for direct content review.

Topics

• Peer Review
• Conference Reviewing
• Reviewer Guidelines
• AI Ethics in Reviewing
• Imposter Syndrome

Best for: AI Researcher, Research Scientist, AI Student

Related on AIssential

• Requesting reduction in reviewer load for NeuRIPS? [D] — NeuRIPS reviewers are typically assigned four papers, with options to request reassignment.
• [D] ICML paper to review is fully AI generated — AI-generated research papers challenge conference policies and reviewer assessment of technical merit and authorship.
• Some Lessons on Reviews and Rebuttals — Effective academic reviewing and rebuttal writing hinge on structured processes and a constructive mindset.
• Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions — AI review can significantly align with human feedback and identify novel issues in paper drafting.
• The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use — The ESE peer review system faces unsustainable workload, quality issues, and ethical dilemmas exacerbated by GenAI, demanding urgent systemic changes.
• Please follow the rules for ARR/ACL papers — Adherence to publication guidelines is crucial for research quality and reviewer efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.