Through the Sentence Lens: Explainable Essay Scoring through Fine-Grained Predictions

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Daniel Mora Melanchthon, Stefan Keller, and Andrea Horbach propose a novel approach to enhance explainability in Automated Essay Scoring (AES) by employing fine-grained predictions at the sentence level. Current AES systems often lack transparency, and existing explainability methods are either counterintuitive or too complex for educational settings. Their method involves using ablation strategies to derive sentence-level pseudo scores from essay-level gold scores, which are then used to train sentence-level models. The researchers evaluated this approach against essay-level baselines on two datasets, ASAP and MEWS, and compared sentence-level output to a human baseline. Findings indicate a trade-off between overall essay-level performance and the granularity achieved at the sentence level. Specifically, for the language quality trait, most sentence-level models achieved performance comparable to the essay-level baseline, while for content, the approach yielded more positive results on prompts with shorter essays.

Key takeaway

For NLP Engineers developing Automated Essay Scoring systems, if you are prioritizing model transparency and pedagogical value, consider implementing fine-grained sentence-level prediction. This approach offers comparable performance for language quality assessment and improved content scoring on shorter essays, enhancing user trust and feedback clarity. You should evaluate the trade-off between overall essay-level performance and the benefits of sentence-level granularity for your specific application.

Key insights

Explainable Automated Essay Scoring can be achieved by training models on sentence-level pseudo scores derived from essay-level gold scores.

Principles

• Fine-grained predictions enhance model transparency.
• Ablation strategies can derive sub-component scores.
• Trade-offs exist between granularity and overall performance.

Method

Sentence-level pseudo scores are derived from essay-level gold scores using ablation strategies. These pseudo scores then train sentence-level models, which are evaluated against essay-level baselines and human baselines.

In practice

• Apply sentence-level scoring for language quality feedback.
• Consider prompt length for content-focused scoring.

Topics

• Automated Essay Scoring
• Explainable AI
• Sentence-level Prediction
• Model Transparency
• Natural Language Processing
• Educational Technology

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models — LLMs encode linearly accessible essay quality representations, identifiable through specific neurons and layer-wise shifts.
• Neuro-symbolic Approaches for Rubric-Based Automatic Essay Evaluation of ENEM Essays — Neuro-symbolic methods can enhance automated essay scoring with interpretability and fine-grained feedback.
• Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework — Linguistic features can quantitatively assess narrative quality, distinguishing professional from self-published works.
• Automated Reformulation of Argumentative Essays to Improve Argument Organization and Development — LLMs can provide targeted, explainable feedback on argumentative essay quality by reformulating text.
• SenFlow: Inter-Sentence Flow Modeling for AI-Generated Text Detection in Hybrid Documents — SenFlow improves AI-generated text detection in hybrid documents by modeling inter-sentence dependencies and using a new benchmark.
• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.