ReproNLP 2026: A Third Replication of the Human Evaluation of a QAG System for Children’s Storybooks

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

ReproNLP 2026 presents a third independent replication of the human evaluation from Yao et al. (2022), which assessed an automated Question-Answer Generation (QAG) system for children's storybooks. The study compared the QAG system against a baseline and human-authored ground truth across Readability, Question Relevance, and Answer Relevance, using five NLP-literate annotators. This replication confirms the original study's main findings: the QAG system outperforms the baseline on Readability and Question Relevance, and Ground Truth consistently ranks highest. System rankings largely persist across all three criteria, with only a statistically non-significant difference in Answer Relevance. Notably, these findings hold despite a severe drop in inter-annotator agreement for Readability. The replication also documents previously unreported methodological concerns, including data quality issues and evaluation design limitations identified during its pilot study.

Key takeaway

For NLP Engineers designing or evaluating Question-Answer Generation systems, you should prioritize robust evaluation methodologies. Be aware that while system performance rankings may replicate, inter-annotator agreement can fluctuate significantly, particularly for subjective criteria like Readability. Your pilot studies must thoroughly identify potential data quality issues and evaluation design limitations to ensure reliable human evaluation outcomes.

Key insights

Human evaluation reproducibility in NLP remains challenging, yet core system performance rankings can persist despite agreement drops.

Principles

• System rankings can be robust.
• Inter-annotator agreement varies.
• Ground truth consistently excels.

Method

This replication involved assessing a QAG system, baseline, and ground truth across three criteria (Readability, Question Relevance, Answer Relevance) using five NLP-literate annotators, including a pilot study to identify methodological issues.

In practice

• Conduct pilot studies thoroughly.
• Monitor inter-annotator agreement.
• Benchmark against human ground truth.

Topics

• Natural Language Processing
• Question-Answer Generation
• Human Evaluation
• Reproducibility
• Inter-annotator Agreement
• Children's Storybooks

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instability — Code-mixed inputs cause significant moderation workflow instability, increasing review burden and non-hate false-flags.
• A Self Consistency Based Reranking for Narrative Question Answering — Semantic agreement among multiple generated answers improves Narrative Question Answering robustness.
• Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale — Human subjectivity annotation is inherently variable and influenced by context, while LLMs offer plausible but distinct scoring.
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews — AI review utility stems from textual justification, not just scalar scores, requiring text-centric evaluation.
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• How Human Feedback Shapes AI-generated Community Notes — Human feedback refines AI-drafted content, improving helpfulness but facing adoption bottlenecks due to participation limits.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.