KEYSCORE — Keystroke-enhanced Automated Essay Scoring

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Education & Learning — Educational Technology (EdTech), Educational Psychology & Learning Sciences, Academic Research & Higher Education · Depth: Expert, medium

Summary

KEYSCORE, a research initiative presented at BEA 2026, explores the predictive power of keystroke logging data for automated essay scoring. Utilizing the newly collected PISA FLA writing process dataset, comprising 3,882 writing sessions, researchers extracted comprehensive keystroke-based features such as temporal measures, pause and burst patterns, deletion behavior, production efficiency, and navigation activity. These features were evaluated for their ability to predict holistic essay scores on a 0–5 scale. The study compared these process-feature models against content-based scoring, including essays written with and without AI chatbot assistance. Analysis revealed that keystroke features offer an early predictive signal, capturing writing fluency and revision aspects before texts are conventionally scorable, suggesting process-based scoring can complement product-based methods for formative, real-time feedback.

Key takeaway

For NLP Engineers developing educational applications, this research suggests integrating keystroke logging data can significantly enhance automated essay scoring. You should consider incorporating process features like temporal measures and deletion patterns to provide formative, real-time feedback to writers, especially early in the drafting process. This approach offers a valuable complement to traditional content-based scoring, enabling more dynamic and responsive assessment tools that capture writing fluency and revision behavior.

Key insights

Keystroke data provides early, genuine signals for automated essay scoring, complementing content-based methods.

Principles

• Writing process features predict essay quality.
• Early keystroke data indicates writing fluency.
• Process-based scoring offers real-time feedback.

Method

Extract keystroke features (temporal, pause, deletion, efficiency, navigation) from writing sessions (e.g., 3,882 PISA FLA sessions) to predict holistic essay scores (0–5 scale), comparing against content-based models.

In practice

• Implement keystroke logging for early writing assessment.
• Develop real-time feedback systems based on process data.
• Integrate process features with content-based scoring.

Topics

• Automated Essay Scoring
• Keystroke Logging
• Writing Process Analysis
• Natural Language Processing
• Educational Technology
• Formative Assessment

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• ScholaWrite: A Dataset of End-to-End Scholarly Writing Process — End-to-end keystroke data with cognitive annotations is crucial for developing advanced, cognitively-aligned AI writing assistants.
• Evaluating Automated Scoring Models on Official ENEM Essays — LLMs pretrained on mock exams significantly improve automated scoring of official ENEM essays, even with small datasets.
• PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback — PsyScore unifies essay scoring and adaptive feedback using a shared psychometric latent ability representation.
• Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs — EEG signals significantly enhance keyphrase extraction from microblogs, complementing eye-tracking data.
• Neuro-symbolic Approaches for Rubric-Based Automatic Essay Evaluation of ENEM Essays — Neuro-symbolic methods can enhance automated essay scoring with interpretability and fine-grained feedback.
• Hemingway-bench Leaderboard: Because Good Writing Isn't a Checklist of Vibes — Current AI writing benchmarks fail to assess true creativity and depth, rewarding superficial adherence or clickbait.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.