[D] ICML 2026: Policy A vs Policy B impact on scores discussion

2026-03-25 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

An informal poll and community discussion at ICML 2026 suggest a potential disparity in paper review scores based on two distinct LLM-review policies. Policy A, a stricter policy prohibiting LLM use, appears to result in harsher average scores (mean 3.23, std dev 0.55, 36 samples) compared to Policy B, which permits limited LLM assistance (mean 3.47, std dev 0.80, 19 samples). Conversely, Policy A reviews show higher reviewer confidence (3.54) than Policy B (3.22). This observation, though based on a small, self-selected sample of 55 responses, aligns with external research indicating that AI-generated reviews tend to be more lenient and positive. The discussion highlights concerns about fairness and the potential for LLM assistance to influence review tone, breadth of knowledge, and perceived novelty.

Key takeaway

For AI Scientists submitting to conferences with varied LLM review policies, be aware that policies allowing LLM assistance may lead to slightly higher average scores and more lenient reviews. If your paper is reviewed under a stricter, no-LLM policy, your scores might appear comparatively harsher. You should advocate for score normalization across different review policy groups to ensure fair evaluation of submissions.

Key insights

LLM-assisted peer review may lead to more lenient scores and polished reviews compared to stricter human-only policies.

Principles

• LLM assistance can bias review scores positively.
• Reviewer confidence may not correlate with score leniency.

Method

An informal poll collected paper scores, review policies, and perceived review harshness/leniency to snapshot community observations on LLM-assisted peer review.

In practice

• Consider LLM policy when interpreting conference scores.
• Evaluate review tone for potential LLM influence.

Topics

• ICML 2026
• LLM-assisted Reviewing
• Peer Review Policies
• Review Score Bias
• Academic Conferences

Best for: AI Scientist, AI Researcher, Research Scientist, AI Student

Related on AIssential

• [D] ICML 2026 Review Discussion — ML peer review suffers from high variance in reviewer quality and increasing submission volumes.
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes — An LLM-based system can accurately predict peer-review outcomes and provide structured feedback for academic manuscripts.
• Seems ICML is rejecting MANY unanimous positively rated papers [D] — ICML 2024 is rejecting many uniformly positively rated papers due to review process distortions and capacity limits.
• Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI — LLMs are making peer reviews longer and more fluent but reducing focus on deep evaluative dimensions.
• Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions — AI review can significantly align with human feedback and identify novel issues in paper drafting.
• Benchmarks in Leipzig — LLMs exhibit impressive and improving mathematical reasoning, solving nearly all research-level math questions in a new benchmark.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.