WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays

2026-07-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

WrAFT, a Writing Assessment and Feedback Tool, offers accurate and reliable scoring alongside comprehensive feedback for argumentative essays. This system employs a modular design, segmenting automated writing evaluation (AWE) into distinct scoring, surface-level feedback, and deep-level feedback components. The development involved evaluating various Large Language Models (LLMs), including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, through both direct prompting and supervised fine-tuning on a proprietary dataset of 480 TOEFL Independent Writing essays. WrAFT achieved a quadratic weighted kappa (QWK) of 0.84 and a root mean square error (RMSE) of 0.44 for scoring on a 0-5 scale. Human evaluation confirmed high approval for feedback, with 96.14 percent for surface-level, 93.03 percent for deep-level macro, and 94.69 percent for deep-level micro feedback. An interactive user interface for WrAFT is publicly available and free to use.

Key takeaway

For educators or NLP engineers developing automated writing evaluation systems, WrAFT demonstrates that a modular LLM-based approach can achieve high accuracy in scoring and provide effective, human-approved feedback. You should consider integrating similar modular LLM strategies to enhance existing AWE systems or improve the quality of feedback provided to students, leveraging diverse LLMs and domain-specific datasets for optimal results.

Key insights

WrAFT's modular LLM-based design delivers high accuracy in essay scoring and human-approved comprehensive feedback.

Principles

• Modular AWE design enhances system clarity.
• LLM fine-tuning improves scoring and feedback quality.
• Proprietary datasets are crucial for domain-specific AWE.

Method

WrAFT divides AWE into scoring, surface, and deep feedback, evaluating LLMs (LLaMA-3.3-70B-Instruct, GPT-4o, Claude 3.7) via direct prompting and supervised fine-tuning on a TOEFL essay dataset.

In practice

• Implement modularity in AWE system architecture.
• Benchmark multiple LLMs for optimal task performance.
• Fine-tune LLMs on specific domain datasets.

Topics

• Automated Writing Evaluation
• Large Language Models
• Argumentative Essays
• TOEFL Writing
• LLaMA-3.3-70B-Instruct
• GPT-4o
• Claude 3.7

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards — RLAES unifies LLM-based essay scoring and feedback generation using reinforcement learning with rubric-based rewards for measurable quality.
• HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing — Tree-of-Writing (ToW) offers a robust, tree-structured method for evaluating LLM writing, correlating highly with human judgment.
• Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System — A multifactor scoring system, integrating five dimensions and a GUI, provides comprehensive evaluation of LLM response quality.
• The Crutch or the Ceiling? How Different Generations of LLMs Shape EFL Student Writings — Advanced LLMs boost surface fluency in EFL writing but may hinder deep learning and mask true student ability.
• Fine-Grained Content Zone Prediction in German Argumentative Essays Using LLMs — Targeted prompt engineering and similarity-based few-shot prompting significantly improve LLM performance for content zone prediction.
• A Three-Level Audit of LLM Alignment for Argument Quality Assessment — Auditing LLM argument quality evaluators requires assessing surface agreement, instructional adherence, and rationale faithfulness for true reliability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.