Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

An evaluation framework for LLM-based Automatic Short Answer Grading (ASAG) is introduced, utilizing Item Response Theory (IRT) to model grading correctness based on latent grader ability and response difficulty. This framework enables a granular, response-level analysis of LLM grading performance, revealing robustness differences not apparent from aggregate metrics like macro-F1 or Cohen's kappa. Applied to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks, the study found that models with similar overall performance can differ substantially in how their accuracy declines with increasing response difficulty. Errors on difficult responses disproportionately concentrated on the "partially_correct_incomplete" label, indicating intermediate-label collapse under ambiguity. Higher response difficulty correlated with weaker semantic alignment to reference answers, stronger contradiction signals, and greater semantic isolation in embedding space.

Key takeaway

For machine learning engineers evaluating LLM performance in Automatic Short Answer Grading, relying solely on aggregate metrics like macro-F1 or Cohen's kappa is insufficient. You should adopt Item Response Theory (IRT) to uncover how LLM grading accuracy varies with response difficulty and identify specific error patterns, especially for ambiguous cases. This will inform more robust model selection and refinement, ensuring your ASAG systems perform reliably across diverse student responses.

Key insights

Item Response Theory provides a granular framework to evaluate LLM ASAG performance beyond aggregate metrics, revealing difficulty-dependent accuracy and error patterns.

Principles

• Aggregate metrics mask LLM grading performance variations.
• LLM grading accuracy declines with increasing response difficulty.
• Ambiguity often leads to "partially_correct_incomplete" errors.

Method

The framework uses Item Response Theory (IRT) to model LLM grading correctness based on latent grader ability and response difficulty, enabling response-level analysis of success and failure points.

In practice

• Evaluate LLM ASAG with IRT for granular insights.
• Analyze difficult responses for semantic alignment and contradiction.
• Monitor "partially_correct_incomplete" for ambiguity collapse.

Topics

• Automatic Short Answer Grading
• Large Language Models
• Item Response Theory
• LLM Evaluation
• Grading Difficulty
• Semantic Alignment

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation — Multilingual-IRT extends Item Response Theory to efficiently evaluate LLMs across languages, identifying translation errors and cultural bias.
• Evaluating Automated Scoring Models on Official ENEM Essays — LLMs pretrained on mock exams significantly improve automated scoring of official ENEM essays, even with small datasets.
• RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering — Automatic metrics for open-ended QA exhibit a validity–discrimination tradeoff, failing to both confirm content alignment and reliably rank systems.
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty — RankLLM offers a difficulty-aware framework for LLM evaluation, jointly quantifying question difficulty and model competency.
• How Uncertain Is the Grade? A Benchmark of Uncertainty Metrics for LLM-Based Automatic Assessment — Categorical uncertainty metrics are generally more effective for LLM-based automatic assessment, while relation-based methods offer superior stability.
• # I Got Tired of Saying “This Response Is Better” Without Being Able to Prove It — Combining human domain expertise with automated pattern detection improves LLM evaluation accuracy and provides objective vocabulary for subjective judgments.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.