ReproHum #0031–01: Reproducing a Human Readability Evaluation for Question–Answer Generation Systems

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

ReproHum #0031–01 presents a reproduction of a human readability evaluation for question–answer generation (QAG) systems, initially performed by Yao et al. (2022). This study, part of the ReproHum project and the ReproNLP 2026 shared task, focused on re-evaluating the readability criterion, one of three from the original work. A new group of five evaluators conducted the assessment, generating descriptive results, inter-annotator agreement metrics, and system-level comparisons. The reproduction's findings consistently support all conclusions of the original evaluation and are largely consistent with two previous reproductions, thereby reinforcing the robustness and understanding of human evaluations in natural language processing system assessment.

Key takeaway

For NLP engineers designing or relying on human evaluations for question–answer generation systems, this reproduction confirms the robustness of such assessments. Your critical evaluations, especially for criteria like readability, can be reliably reproduced by different evaluator groups, validating initial findings. Consider incorporating reproducibility checks into your evaluation pipeline to strengthen confidence in system comparisons and performance claims.

Key insights

Human evaluations for NLP systems, particularly readability, demonstrate robust reproducibility across different evaluators.

Principles

• Reproducing human evaluations validates NLP system assessments.
• Inter-annotator agreement is key for evaluation robustness.

Method

Replicated a human readability evaluation for QAG systems using a new group of five evaluators, then compared results and robustness metrics to original and prior reproductions.

In practice

• Conduct reproducibility studies for critical human evaluations.
• Compare results with multiple evaluator groups.

Topics

• Natural Language Processing
• Human Evaluation
• Reproducibility
• Question-Answer Generation
• Readability Assessment
• NLP System Evaluation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews — AI review utility stems from textual justification, not just scalar scores, requiring text-centric evaluation.
• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• The Inadequacy of Automatic Evaluation Metrics in Question Answering: A Case-Study in Portuguese — Traditional lexical metrics are insufficient for evaluating Question Answering quality, especially in non-English languages.
• Evaluating AI Humanizers: Accuracy, Privacy, and Real-World Performance — Effective AI humanizers rewrite machine-generated text to evade detection, requiring rigorous, independent evaluation focused on accuracy, privacy, and real-world performance.
• Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025 — NLP human annotation reporting often lacks critical validity details, hindering reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.