Predicting Item Difficulty and Generating Reading Comprehension Items via an Annotated Repository

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study developed a penalized regression model to predict item difficulty for reading comprehension passages, recovering IRT-based difficulty from p-values. The model utilizes an annotated repository of reading passages and student data from US standardized tests (New York and Texas, grades 3-8, 2018-23). This repository includes meta-data on linguistic, test, and context features. The model achieved an RMSE of 0.59, significantly outperforming a 0.92 baseline, and demonstrated a 0.77 correlation between true and predicted difficulty. Evaluation of LLM embeddings (ModernBERT, BERT, and LLaMA) showed marginal performance improvements, but confirmed their effectiveness as standalone alternatives to traditional linguistic features. This difficulty prediction model now powers a publicly available, human-in-the-loop tool for generating reading comprehension items.

Key takeaway

For educational technologists or NLP engineers developing assessment tools, this research indicates that robust item difficulty prediction is achievable using annotated historical test data. You should consider implementing penalized regression models, potentially integrating LLM embeddings like ModernBERT or LLaMA as efficient feature sets. This approach can power automated, human-in-the-loop systems to generate more precisely calibrated reading comprehension items, enhancing test development efficiency and quality.

Key insights

A penalized regression model accurately predicts reading comprehension item difficulty using annotated test data and powers an item generation tool.

Principles

• IRT difficulty can be recovered from p-values.
• Annotated meta-data improves difficulty prediction.
• LLM embeddings are effective standalone features.

Method

Item difficulty is modeled using a penalized regression on an annotated repository of reading passages and student data, incorporating linguistic, test, and context features.

In practice

• Use penalized regression for difficulty prediction.
• Integrate LLM embeddings as feature alternatives.
• Develop human-in-the-loop item generation tools.

Topics

• Item Difficulty Prediction
• Reading Comprehension
• Penalized Regression
• LLM Embeddings
• Automated Item Generation
• Educational Assessment

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning — Difficulty-aware data partitioning for SFT-then-RL training significantly enhances LLM general reasoning.
• What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction — Soft-target loss fine-tuning significantly improves LLM and MLM performance in continuous value prediction tasks.
• Evaluating Automated Scoring Models on Official ENEM Essays — LLMs pretrained on mock exams significantly improve automated scoring of official ENEM essays, even with small datasets.
• Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation — Explicitly modeling MCQ distractor structure significantly improves automatic difficulty estimation.
• Concept Extraction and Webb’s Depth of Knowledge: Comparing LLM Question Generation Pipelines for Educational Assessment — Comparing LLM pipelines for educational question generation using concept extraction and Webb's Depth of Knowledge.
• LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment — LLMs struggle to reliably measure item discrimination, a key psychometric property for distinguishing student proficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.