I am worried by NLP research culture

2026-06-08 · Source: Ehud Reiter's Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Intermediate, medium

Summary

The NLP research culture in 2026 is described as significantly worse than in 1990, primarily due to a shift from prioritizing scientific findings to focusing on publication counts for career advancement. This problematic culture manifests in several ways, including a lack of scientific rigor, evidenced by the use of poor datasets, meaningless or obsolete benchmarks like ROUGE, and insufficient support for reproducibility. The author also points to poorly executed experiments, major data contamination issues, minimal interest in real-world impact, and an unhealthy fixation on beating leaderboards. Furthermore, the community faces a rapidly increasing problem of cheating and fraud, with ACL in 2026 desk-rejecting 100 submissions for hallucinated citations. Finally, the culture is criticized for its lack of openness to new ideas and people, exhibiting a strong bias towards trendy LLM-based research and creating barriers for researchers without funding, with caring responsibilities, disabilities, or those unfamiliar with LaTeX.

Key takeaway

For AI scientists and students navigating NLP research, recognize that current cultural pressures often prioritize publication volume over scientific integrity. You should actively resist "corner cutting" by ensuring your experiments are rigorous, using meaningful benchmarks, and striving for reproducibility. Prioritize genuine scientific contribution over leaderboard chasing. Additionally, be vigilant against fraud and advocate for a more open community that values diverse methodologies and researchers, rather than solely focusing on trendy LLM approaches.

Key insights

NLP research culture in 2026 prioritizes publication volume over scientific rigor, fostering fraud and limiting openness.

Principles

• Research culture dictates scientific behaviors.
• Prioritize scientific rigor over publication counts.
• Strong culture discourages fraud and promotes openness.

In practice

• Conduct rigorous experiments.
• Avoid cheating and fraud.
• Embrace diverse research approaches.

Topics

• NLP Research Culture
• Scientific Rigor
• Research Fraud
• Reproducibility
• LLM Bias
• Academic Publishing

Best for: Research Scientist, AI Scientist, AI Student

Related on AIssential

• The uncritical adoption of AI in science is alarming — we urgently need guard rails — Uncritical AI adoption in science risks reduced research quality, narrowed scope, and deskilling future scientists.
• Greater than 80% of researchers at CVPR are chinese. This speak volumes on the chinese nexus in research, and something needs to be done about it. [D] — The integrity of machine learning conference peer review is widely perceived as compromised by systemic biases and quality issues.
• The social sciences need tools for the 21st century | Letters — Social science needs new tools and methods to improve replicability and advance understanding of human behavior.
• Attributing and situating knowledge cannot be left to language models — LLMs cannot reliably handle scholarly attribution, risking miscitation and undermining research standards.
• Will AI ruin the social sciences — or revolutionize them? — Large language models are simultaneously compromising social science research data integrity and offering tools for enhanced analytical robustness.
• 142 - Science Of Science, with Kyle Lo — SciSci uses computational methods to study scientific processes, guiding tool development for community-wide benefit.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Ehud Reiter's Blog.